Kv-cache streaming for improved performance and fault tolerance in generative model serving

Abstract

A method of serving a generative transformer model includes determining a batch size to use in processing inference requests and allocating at least one prompt pipeline and at least on token pipeline to the generative transformer model to process the batch of inference requests. The number of prompt pipelines and the number of token pipelines, and the depths of the pipelines are determined based on the batch size, an average prompt length, a cache requirement per stage, and a memory footprint of model weights for the generative model per stage using a resource allocator component of the model serving system. Cache streaming is used to stream prompt cache from prompt pipelines to token pipelines to generate tokens. Cache streaming involves gather-copy operations which may be performed using compute kernels.