Query response interface with server side generative model(s)

Abstract

Various implementations include processing, at a client device, an instance of audio data capturing a user voice query using an automatic speech recognition model to generate a sequence of instances of tokenizable query text. In many implementations, one or more instances of the sequence can be transmitted to a remote computing system prior to generating the entire sequence. In a variety of implementations, each instance in the sequence can be processed using a generative model which includes a streaming multi-head attention portion. Responsive output can be transmitted from the remote computing system to the client device, where the client device renders the responsive output to the user. In many implementations, the time between the user speaking the user query and the client device rendering the responsive output is reduced, thus decreasing latency in the system.