Systems and methods for heterogeneous large language model prompt attention-processing

Abstract

Methods and systems are disclosed for implementing a Large Language Model utilizing a prompt attention-processing subsystem and a generation attention-processing subsystem. A sequence of tokens is first processed by a prompt attention-processing subsystem, which utilizes an associated prompt KV-cache to store matrix values generated during prompt attention-processing. Upon the completion of prompt attention-processing, the populated prompt KV-cache is transferred to a generation KV-cache for processing by the generation attention-processing subsystem. The prompt and generation attention-processing subsystem can be multi-headed. The separate processing of the prompt facilitates efficient computations. Further, the prompt can be processed in segments that match available memory and computational resources. The generation attention-processing subsystem then produces an output token sequence based on the prompt KV-cache values transferred to the generation attention-processing system. The described system ensures optimized processor and memory usage and streamlined processing for large language model systems.