Fine-tuning large language model(s) using reinforcement learning with search engine feedback

Abstract

Various implementations are directed towards fine-tuning a large language model (LLM) using search engine feedback (e.g., responsive content generated based on a reference source material such as a set of search engine results). Additionally or alternatively, a supervision signal can be generated based on comparing search engine conditioned LLM output with unconditioned LLM output. In many implementations, the supervision signal(s) can be used in training a reward model using reinforcement learning, where the trained reward model can be used in fine-tuning the LLM.