Amazon Technologies, Inc.

Mozilla Corporation

Xiph.org Foundation

Commonwealth Scientific and Industrial Research Organisation

Jean-Marc Valin

Timothy B Terriberry

Karim Helwani

Arvindh Krishnaswamy

Mehmet Umut Isik

Ritwik Giri

Michael Mark Goodwin

Neerad Dilip Phansalkar

Jan Buethe

Ahmed Mustafa

Paris Smaragdis

Daniel Ryan

Iain Bruce Collings

Richard Newman

Srikanth Venkata Tenneti

Erfan Soltanmohammadi

Masahito Togami

Shrikant Venkataramani

Neural coding techniques may be implemented for transmission of redundant audio data. An encoding technique is implemented that uses forward encoding along with an initial state to include multiple audio frames from audio data represented as latent vectors in a network packet transmitted to a recipient. The recipient can then use backward decoding and the initial state to obtain the multiple audio frames from the latent vectors. The multiple audio frames provided in the network packet are redundant audio data that can be used to generate missing audio data.

Neural coding for redundant audio information transmission

Efficient voice synthesis using frame-based processing may be performed. An audio processing system converts an input speech waveform to an acoustic feature representation, which includes a sequence of frames at a lower resolution than the sampling resolution of the input waveform. The system propagates the acoustic feature representation through GRUs and fully-connected layers, while maintaining the lower resolution. At the end, the system performs a flattening operation on the frames of the final acoustic feature representation to generate an output waveform at a target sampling resolution.

Efficient voice synthesis using frame-based processing

Real-time audio enhancement for a target speaker may be performed. An embedding of a sample of speaker audio is created using a trained neural network that performs voice identification. The embedding is then concatenated with the input features of a trained machine learning model for audio enhancement. The audio enhancement model can recognize and enhance a target speaker's speech in a real-time implementation, as the embedding is in the same feature space of the audio enhancement model.

Real-time target speaker audio enhancement

Real-time low-complexity stereo speech enhancement with spatial cue preservation may be performed. A stereo speech enhancement system receives a stereo input signal (e.g., a left and right input signal). The stereo speech enhancement system estimates spatial cues for a target speaker and downmixes the stereo input signal into a monaural signal. A low-complexity model may then process the monaural signal to generate an enhanced monaural signal. The stereo speech enhancement system upmixes the enhanced monaural signal based on the estimated spatial cues for the target speaker, to generate an enhanced stereo output signal.

Real-time low-complexity stereo speech enhancement with spatial cue preservation

Techniques for training and using a machine learning model for estimation of reverberation in a multi-task learning framework are described. According to some embodiments, the multi-task learning framework improves the performance of the machine learning model by estimating the amount of reverberation present in an input audio recording as a secondary task to the primary task of generating a clean speech portion of the input audio recording. In one embodiment, a model architecture is selected that takes a noisy reverberant recording as an input and outputs an estimate of a clean (e.g., de-reverberated) signal, an estimate of noise (e.g., background noise), and an estimate of the reverb only portion, with the secondary task of estimating the reverb only portion acting as a regularizer that improves the machine learning model's performance in enhancing the reverberant (e.g., and noisy) input speech.

Speech enhancement machine learning model for estimation of reverberation in a multi-task learning framework

Audio processing may be performed with a convolutional neural network that includes positional embeddings. Audio data may be received at an audio processing system. A convolutional neural network that concatenates frequency-positional embeddings at an input layer may be used to process the audio data. A result of processing the audio data through the convolutional neural network may be used to perform an audio processing task.

Convolutional neural network with positional embeddings for audio processing

Joint noise and echo suppression may be performed for enhancing two-way audio communications. Audio data is captured at a communication device and audio data transmitted to the communication device from another communication device are used as input features to a trained machine learning model that uses the transmitted audio data as a reference signal to eliminate residual echo in the captured audio data when also suppressing noise in the captured audio data.

Joint noise and echo suppression for two-way audio communication enhancement

Post-filtering may be performed for ratio masks as part of audio enhancement. Audio data may be received. A machine learning model may be applied to generate gain values for different spectrum bands of the audio data. The gain values may then be modified using an envelope post-filter according to a monotonically increasing function applied to the gain values to produce modified gain values used to generate an enhanced version of the audio data.

Ratio mask post-filtering for audio enhancement

Systems and methods are described for removing ringing artifacts from a coded image. For each block of coded image data, a direction is identified by selecting a directional block divided into a plurality of pixel lines in one predetermined direction of a set of at least four predetermined directions. The directional block may have a constant value across each pixel line. A parameter related to a sum of a mean-square difference between a pixel value and the pixel average of pixels falling on a pixel line in the directional block that includes the location of the pixel may be calculated for each direction of the set. The direction of the set having a minimum summed mean-square difference is selected as the direction for the block based on the calculated parameter values. A non-linear filter is then applied to each block based on the identified direction for the block.

Directional deringing filters

Systems and methods are described for predicting a mode value of a block of an image. Mode values are received for a predetermined number of neighboring blocks of the image. Each mode value may within a predetermined range of integer values. Using a probability table, a probability of each mode value in the range of integers being selected may be computed. The probability table may have a first axis that includes every integer value within the range of integer values, and second axis that includes combinations of the neighboring blocks, each combination of the neighboring blocks representing a binary determination whether or not each neighboring block is equal to a corresponding integer value on the first axis. Based on the computed probabilities, a mode value for the block may be selected and coded by an entropy coder.

Probability modeling of intra prediction modes

An apparatus and corresponding method are provided for pyramid vector quantization of video data, including receiving the video data in the frequency domain; and pyramid vector quantizing at least one group of pictures (GOP) from the received video data in accordance with a potentially varying bandwidth, the GOP comprising key frames and predicted frames.

Pyramid vector quantization for video coding

Systems and methods are described for encoding quantized vector parameters in a bitstream are described. An exemplary method may include receiving a vector of integers used in a data compression codebook, the sum of the integers equaling a pulse sum. An initial expected magnitude may be determined for a first integer, the initial expected magnitude being based on the pulse sum, a distribution parameter, and a value corresponding to a number of integers in the vector. The actual magnitude of the first integer may be encoded based on the initial expected magnitude of the first integer. The pulse sum may be adjusted using the encoded actual magnitude. Also, the value corresponding to the number of integers in the vector may be reduced by one. Expected magnitudes for each of the remaining integers of the vector may then be calculated recursively.

Vector quantization with non-uniform distributions

Embodiments are described of a multi-block coding scheme for an audio signal to prevent partial collapse conditions from causing pre-echo compression artifacts. An audio codec includes a segmentation component partitioning the audio signal into a plurality of tiles, wherein each tile comprises data from a particular segment of time and a particular set of frequencies of the audio signal; a band energy component determining an energy value for each tile corresponding to a signal component in a respective tile; an encoder flag tracking component marking a tile as not collapsed or collapsed based on the energy value in that tile; and a decoder flag tracking component filling all tiles marked as collapsed with pseudorandom noise at an estimated energy level.

Methods and systems for avoiding partial collapse in multi-block audio coding

Embodiments are described for a system and method for implementing an adaptive time-frequency resolution in audio and video coding systems. A method of adaptively transforming the time-frequency resolution for a defined spectrum comprises dividing the spectrum of the input signal into a into plurality of bands; determining, for each band of the plurality of bands, a characteristic of the content (e.g., tonal or transient content); modifying the time-frequency resolution value to one or more bands of the plurality of bands to increase either a time resolution of the band or a frequency resolution of the band depending on the characteristic of the content; determining a cost associated with modifying the time-frequency resolution value of the one more bands based on an entropy measure of the bands, and altering the modified time-frequency resolution values in a manner that accounts for the coding cost.

Methods and systems for adaptive time-frequency resolution in digital data coding

Embodiments are generally directed to systems and methods for bit allocation and band partitioning for gain-shape vector quantization in an audio codec. An audio codec implements a method that uses an implicit, dynamic scheme to allow an encoder and decoder to recreate a series of bit allocation decisions for gain and shape without transmitting additional side information for each decision, based on the number of bits that are left remaining and available in a given packet. For implementation in practical codecs, the band comprising the allocation of bits for the shape is recursively split into equal partitions until the number of bits allocated to each partition is less than the maximum codebook size.

Growing community of inventors

Montreal, Canada

Jean-Marc Valin