Midstream processing of streaming input to generate streaming output

Abstract

Implementations are described herein for processing a stream of time-varying input data to generate/predict a stream of time-varying output data in real-time or near-real time. In various implementations, while a stream of input frames, such as a stream of audio input frames, is received, audio input frames received up to a current time step may be tokenized (e.g., midstream) to generate a stream of audio input tokens. A Transformer-based causal attention model may be used to predict a stream of audio output tokens, e.g., by iteratively applying the Transformer-based causal attention model to: at least some of the audio input tokens tokenized up to the current time step, and at least some of the audio output tokens predicted up to the current time step. The stream of audio output tokens may be detokenized to generate a stream of audio output frames.