Method and system for real-time and low latency synthesis of audio using neural networks and differentiable digital signal processors

Abstract

Example aspects include techniques for implementing real-time and low-latency synthesis of audio. These techniques may include generating a frame by sampling audio input in increments equal to a buffer size of until a threshold corresponding to a frame size used to train a machine learning (ML) model is reached, detecting feature information within the frame, determining, by the ML model, control information for audio reproduction based on the feature information. In addition, the techniques may include generating filtered noise information by inverting the noise magnitude control information using an overlap and add technique, generating, based on the control information, additive harmonic information by combining a plurality of scaled wavetables, and rendering audio output based on the filtered noise information and the additive harmonic information.