System and method for deep large language model thinking with gradient feedback alignment

Abstract

A large language model (LLM) is refined using an unsupervised alignment process for enhanced reasoning and deep thinking capabilities. An iterative search and optimization procedure is used to explore the space of possible thought generations enabling the model to improve deep thinking capabilities without direct supervision. The LLM is prompted to produce a first thought and response pair in response to a query. The thought and response pair is evaluated with a trained judge LLM, which provides feedback for the thought and for the response. A revised prompt is generated in response to the feedback and provided to the LLM, which produces second thought and response pair. The LLM is refined based on the first, non-preferred, thought and response pair and the second, preferred, thought and response pair to align with the preferred thought and response pair, e.g., using a direct preference optimization process.