Dolby Laboratories Licensing Corporation

Dolby International Ab

Dublin International Ab

Lie Lu

Lianwu Chen

Aaron Steven Master

Alan Jeffrey Seefeldt

Ming Wang

Nicolas R Tsingos

Jundai Sun

Heiko Purnhagen

Xuejing Sun

Zhiwei Shuang

Dirk Jeroen Breebaart

Ziyu Yang

Antonio Mateos Sole

Harald Mundt

Regunathan Radhakrishnan

Chen Zhang

Rhonda J Wilson

Joshua Brandon Lando

Claus Bauer

Mingqing Hu

Xin Liu

David Matthew Fischer

Heidi-Maria Lehtonen

Guiping Wang

Poppy Crum

Qingyuan Bin

Jia Dai

Chunmao Zhang

Jeremy Grant Stoddard

Antonio Mateos Solé

Shaofan Yang

Daniel Arteaga

Karl Jonas Roeden

Alex Cabrer Manning

Toni Hirvonen

Eduard Aylon Pla

Systems, methods, and computer program products for audio processing based on convolutional neural network (CNN) are described. The CNN architecture may comprise a multi-scale input block and a multi-scale nested block. The multi-scale input block may be configured to receive input data and to generate a first downsampled input data set by downsampling the input data. The multi-scale nested block may comprise a first encoding layer configured to generate a first encoded data set by performing a convolution based on the input data. The multi-scale nested block may comprise a second encoding layer configured to generate a second encoded data set by performing a convolution based on the first downsampled input data set. Furthermore, the multi-scale nested block may comprise a first convolutional layer configured to generate a first output data set by upsampling the second encoded data set, concatenating the first encoded data set and the upsampled second encoded data set, and performing a convolution. The first convolutional layer may be nested between the encoding layers and decoding layers, thereby increasing the number of communication channels with the CNN and simplifying the underlying optimization problem.

Method and appartus for audio processing using a nested convolutional neural network architechture

Computer-implemented methods and devices for combined audio separation and classification are provided. An estimated separated signal is time gated based on a determination of an audio classifier of, at least in part, the original mix of signals before separation. Combined separation, classification, and time gating of both the estimated signal and a residual signal are also provided.

Method and apparatus combining separation and classification of audio signals

A method comprises: obtaining softmask values for frequency bins of time-frequency tiles representing an audio signal; reducing, or expanding and limiting, the softmask values; and applying the reduced, or expanded and limited, softmask values to the frequency bins to create a time-frequency representation of an estimated target source. An alternative method comprises, for each time-frequency tile: obtaining softmask values; applying the softmask values to the frequency bins to create a time-frequency domain representation of an estimated target source; obtaining a panning parameter and a source concentration estimates for the target source; determining, using the panning parameter estimate and the softmask values, a magnitude for the time-frequency representation of the estimated target source; determining, using the panning parameter estimate and the source phase concentration estimate, a phase for the time-frequency representation of the estimated target source; and combining the magnitude and the phase.

Perceptual optimization of magnitude and phase for time-frequency and softmask source separation systems

A method of audio content identification includes using a two-stage classifier. The first stage includes previously-existing classifiers and the second stage includes a new classifier. The outputs of the first and second stages calculated over different time periods are combined to generate a steering signal. The final classification results from a combination of the steering signal and the outputs of the first and second stages. In this manner, a new classifier may be added without disrupting existing classifiers.

Audio content identification

In an embodiment, a method comprises: transforming one or more frames of a two-channel time domain audio signal into a time-frequency domain representation including a plurality of time-frequency tiles, wherein the frequency domain of the time-frequency domain representation includes a plurality of frequency bins grouped into subbands. For each time-frequency tile, the method comprises: calculating spatial parameters and a level for the time-frequency tile; modifying the spatial parameters using shift and squeeze parameters; obtaining a softmask value for each frequency bin using the modified spatial parameters, the level and subband information; and applying the softmask values to the time-frequency tile to generate a modified time-frequency tile of an estimated audio source. In an embodiment, a plurality of frames of the time-frequency tiles are assembled into a plurality of chunks, wherein each chunk includes a plurality of subbands, and the method described above is performed on each subband of each chunk.

Methods, apparatus, and systems for detection and extraction of spatially-identifiable subband audio sources

An audio processing system and method which calculates, based on spatial metadata of the audio object, a panning coefficient for each of the audio objects in relation to each of a plurality of predefined channel coverage zones. Converts the audio signal into submixes in relation to the predefined channel coverage zones based on the calculated panning coefficients and the audio objects. Each of the submixes indicating a sum of components of the plurality of the audio objects in relation to one of the predefined channel coverage zones. Generating a submix gain by applying an audio processing to each of the submix and controls an object gain applied to each of the audio objects. The object gain being as a function of the panning coefficients for each of the audio objects and the submix gains in relation to each of the predefined channel coverage zones.

Processing object-based audio signals

Diffuse or spatially large audio objects may be identified for special processing. A decorrelation process may be performed on audio signals corresponding to the large audio objects to produce decorrelated large audio object audio signals. These decorrelated large audio object audio signals may be associated with object locations, which may be stationary or time-varying locations. For example, the decorrelated large audio object audio signals may be rendered to virtual or actual speaker locations. The output of such a rendering process may be input to a scene simplification process. The decorrelation, associating and/or scene simplification processes may be performed prior to a process of encoding the audio data.

Method, apparatus or systems for processing audio objects

Systems and methods for preserving headphone rendering mode (HRM) in object clustering are described. In an embodiment, an object-based audio data processing system includes a processor configured to receive a plurality of audio objects, wherein an audio object of the plurality of audio objects is associated with respective object metadata that indicates respective spatial position information and an HRM; determine a plurality of cluster positions by applying an extended hybrid distance metric to a spatial coding algorithm to calculate a partial loudness for each of the audio objects; render the audio objects to the cluster positions to form a plurality of clusters by applying the extended hybrid distance metric to the spatial coding algorithm to calculate object-to-cluster gains; and transmit the clusters to a spatial reproduction system.

Headphone rendering metadata-preserving spatial coding

Volume leveler controller and controlling method are disclosed. In one embodiment, A volume leveler controller includes an audio content classifier for identifying the content type of an audio signal in real time; and an adjusting unit for adjusting a volume leveler in a continuous manner based on the content type as identified. The adjusting unit may configured to positively correlate the dynamic gain of the volume leveler with informative content types of the audio signal, and negatively correlate the dynamic gain of the volume leveler with interfering content types of the audio signal.

Volume leveler controller and controlling method

The present application relates to a method of extracting audio features in a dialog detector in response to an input audio signal, the method comprising dividing the input audio signal into a plurality of frames, extracting frame audio features from each frame, determining a set of context windows, each context window including a number of frames surrounding a current frame, deriving, for each context window, a relevant context audio feature for the current frame based on the frame audio features of the frames in each respective context, and concatenating each context audio feature to form a combined feature vector to represent the current frame. The context windows with the different length can improve the response speed and improve robustness.

Dialog detector

Described herein is a method for Convolutional Neural Network (CNN) based speech source separation, wherein the method includes the steps of: (a) providing multiple frames of a time-frequency transform of an original noisy speech signal; (b) inputting the time-frequency transform of said multiple frames into an aggregated multi-scale CNN having a plurality of parallel convolution paths; (c) extracting and outputting, by each parallel convolution path, features from the input time-frequency transform of said multiple frames; (d) obtaining an aggregated output of the outputs of the parallel convolution paths; and (e) generating an output mask for extracting speech from the original noisy speech signal based on the aggregated output. Described herein are further an apparatus for CNN based speech source separation as well as a respective computer program product comprising a computer-readable storage medium with instructions adapted to carry out said method when executed by a device having processing capability.

Method and apparatus for speech source separation based on a convolutional neural network

Example embodiments disclosed herein relate to audio object clustering. A method for metadata-preserved audio object clustering is disclosed. The method comprises classifying an audio object into at least a category based rendering mode information metadata. The method further comprises assigning a predetermined number of clusters to the categories and rendering the audio object based on the rendering mode. Corresponding system and computer program product are also disclosed.

Metadata-preserved audio object clustering

A method of processing audio content including a plurality of audio elements comprises: clustering the plurality of audio elements into a plurality of clusters of audio elements; and for a cluster among the plurality of clusters: for each audio element in the cluster, determining a measure of energy that the audio element contributes to the cluster; for at least one audio element in the cluster, determining a compensation gain based at least in part on the measures of energy for the audio elements in the cluster; and applying the compensation gain to the at least one audio element in the cluster.

Adaptive loudness normalization for audio object clustering

An apparatus and method of blind detection of binauralized audio. If the input content is detected as binaural, a second binauralization may be avoided. In this manner, the user experience avoids audio artifacts introduced by multiple binauralizations.

Blind detection of binauralized stereo content

Example embodiments disclosed herein relate to perception based multimedia processing. There is provided a method for processing multimedia data, the method includes automatically determining user perception on a segment of the multimedia data based on a plurality of clusters, the plurality of clusters obtained in association with predefined user perceptions and processing the segment of the multimedia data at least in part based on determined user perception on the segment. Corresponding system and computer program products are disclosed as well.

Growing community of inventors

Dublin, CA, United States of America

Lie Lu