Speculative decoding techniques for generative models

Abstract

Techniques for improving speculative decoding for improved compute utilization are described. A system may determine a compute budget representing a maximum number of speculative tokens to be verified based on hardware properties, a batch size and an input length. Compute budgets for varying batch sizes and input lengths may be determined offline. At inference time, the compute budget corresponding to a runtime batch size and runtime input lengths may be used to determine the maximum number of tokens that can be verified by a generative model while maintaining efficient compute utilization. The compute budget may be allocated/distributed among the inputs of the batch to efficiently process the inputs in parallel. The allocated budget may be updated for subsequent timesteps. The compute budget may represent the maximum number of tokens that can be verified without the system processing becoming a compute-bound scenario.