Baidu USA LLC

Hee Woo Jun

Gregory Diamos

Joel Hestness

Sercan Omer Arik

Kenneth W Church

Sanjeev Satheesh

Yanqi Zhou

Anuroop Sriram

Sharan Narang

Eric Undersander

Yashesh Gaur

Newsha Ardalani

Md Mostofa Ali Patwary

As deep learning application domains grow, a deeper understanding of the relationships between training set size, computational scale, and model accuracy improvements is extremely beneficial. Presented herein are large-scale empirical study of error and model size growth as training sets grow. Embodiments of a methodology for this measurement are introduced herein as well as embodiments for predicting other metrics, such as compute-related metrics. It is shown herein that power-law may be used to represent deep model relationships, such as error and training data size. It is also shown that model size scales sublinearly with data size. These scaling relationships have significant implications on deep learning research, practice, and systems. They can assist model debugging, setting accuracy targets, and decisions about data set growth. They can also guide computing system design and underscore the importance of continued computational scaling.

Predicting deep learning scaling

For the problem of waveform synthesis from spectrograms, presented herein are embodiments of an efficient neural network architecture, based on transposed convolutions to achieve a high compute intensity and fast inference. In one or more embodiments, for training of the convolutional vocoder architecture, losses are used that are related to perceptual audio quality, as well as a GAN framework to guide with a critic that discerns unrealistic waveforms. While yielding a high-quality audio, embodiments of the model can achieve more than 500 times faster than real-time audio synthesis. Multi-head convolutional neural network (MCNN) embodiments for waveform synthesis from spectrograms are also disclosed. MCNN embodiments enable significantly better utilization of modern multi-core processors than commonly-used iterative algorithms like Griffin-Lim and yield very fast (more than 300&#xd7; real-time) waveform synthesis. Embodiments herein yield high-quality speech synthesis, without any iterative algorithms or autoregression in computations.

Spectrogram to waveform synthesis using convolutional networks

Described herein are systems and methods for word embeddings to avoid the need to throw out rare words appearing less than a certain number of times in a corpus. Embodiments of the present disclosure involve group words into clusters/classes for multiple times using different assignments of the vocabulary words to a number of classes. Multiple copies of the training corpus are then generated using the assignments to replace each word with the appropriate class. A word embedding generating model is run on the multiple class corpora to generate multiple class embeddings. An estimate of the gold word embedding matrix is then reconstructed from multiple pairs of assignments, class embeddings, and covariances. Test results show the effectiveness of embodiments of the present disclosure.

Embeddings with classes

Described herein are systems and methods for a general, scalable, end-to-end framework that uses a generative adversarial network (GAN) objective to enable robust speech recognition. Encoders trained with the proposed approach enjoy improved invariance by learning to map noisy audio to the same embedding space as that of clean audio. Embodiments of a Wasserstein GAN framework increase the robustness of seq-to-seq models in a scalable, end-to-end fashion. In one or more embodiments, an encoder component is treated as the generator of GAN and is trained to produce indistinguishable embeddings between labeled and unlabeled audio samples. This new robust training approach can learn to induce robustness without alignment or complicated inference pipeline and even where augmentation of audio data is not possible.

Systems and methods for robust speech recognition using generative adversarial networks

Growing community of inventors

Sunnyvale, CA, United States of America

Hee Woo Jun