Template guided speculative decoding

Abstract

Aspects of the present disclosure involve accelerated machine learning based sequence generation. Aspects include generating a first portion of a sequence based on an input query using a speculative decoding process involving a draft machine learning model and a target machine learning model with a larger number of parameters than the draft model. Aspects include determining, based on the input query and the first portion, that a next portion of the sequence is predicted to correspond to a template. Aspects include, based on the determining, generating the next portion of the sequence, without using the draft machine learning model, through a template-based generative process involving one or more rules or a generative machine learning model having fewer parameters than the draft machine learning model. Aspects include verifying the next portion using the target machine learning model and appending the next portion to the sequence based on the verifying.