Kv cache block-quantization oriented data handling

Abstract

A computing system for performing runtime data handling optimization for generative models is provided. The computing system comprises at least one processor and memory comprising a first memory and a second memory, wherein the memory stores instructions that, when executed by the at least one processor, cause the at least one processor to execute a generative model. The computing system computes a first value matrix entry based upon the processing of an input to the generative model. The first value matrix entry is stored in a first memory wherein a first group of value matrix entries is identified. The computing system executes data quantization on the first group of value matrix entries which results in a first quantized value matrix. The first quantized value matrix is added to a second memory where it can be used during generation of the generative model. When the first group of value matrix entries is less than a group size parameter, data padding matrix values are generated and used during execution of the data quantization.