Summarizing context and conditioning a large language model to generate responses based on summarized context

Abstract

The present disclosure relates to systems, non-transitory computer-readable media, and methods for reducing context size for a large language model (LLM) by generating context summaries. In particular, the present disclosure reduces context size and improves the accuracy of existing LLMs by generating and storing context summaries. In particular, the disclosed systems utilize an LLM to determine what context to summarize and when to summarize the context as a conversation progresses. The disclosed systems further update model context to include context summaries instead of full session transcripts or model reasoning. Furthermore, in some implementations, the disclosed systems can use a re-materialization prompt to trigger the LLM to recall particular portions of session transcripts or model reasoning to condition the LLM. Thus, the system can use context summaries to fill context windows for informing LLM response generation as opposed to storing memory-intensive conversation transcripts and model reasoning.