Generating draft sequence rankings for speculative decoding using large language model hidden states

Abstract

The present disclosure relates to systems, non-transitory computer-readable media, and methods for using hidden states of a large language model to generate responses to queries via speculative decoding. In particular, the disclosed systems determine, for a large language model ('LLM'), a prompt comprising a query and an input guide text related to the query. The disclosed systems determine, at a time step of the LLM, candidate draft predictions from token sequences in the input guide text that correspond to a most recent token generated by the LLM. The disclosed systems generate, at the time step of the LLM, a draft prediction by comparing tokens associated with the candidate draft predictions to a hidden state of a previous token prior to the most recent token. Furthermore, the disclosed systems generate, for display via a client device, a response comprising the draft prediction for the prompt to the LLM.