Incremental context composition for latency reduction in language processing machine learning models

Abstract

Aspects of the present disclosure involve reducing latency in generating responses using a large language model (LLM). Aspects include applying a sentence boundary detection technique to an input buffer in order to identify a first sentence in input text while the input text is being input. Aspects include using an encoder portion of the LLM to generate a first embedding of the first sentence. Aspects include determining that a final sentence has been entered in the input text. Aspects include using the encoder portion of the LLM to generate a second embedding of the final sentence. Aspects include generating a cumulative context vector using a merging machine learning model based on the first embedding and the second embedding. Aspects include injecting the cumulative context vector into a decoder portion of the LLM. Aspects include receiving a response to the input text from the LLM based on the cumulative context vector.