Baidu USA LLC

Markus Kliegl

Adam Coates

Christopher Fougner

Rewon Child

Sercan Omer Arik

Ryan Prenger

Sanjeev Satheesh

Zhenyao Zhu

Andrew Gibiansky

Eric Dean Battenberg

Hairong Liu

Anuroop Sriram

Vinay Rao

Joel Hestness

Mohammad Shoeybi

Kexin Zhao

Heewoo Jun

Yashesh Gaur

Ajay Kannan

Jiaji Huang

Atul Kumar

David Seetapun

Siddharth Goyal

Kavya Srinet

Described herein are systems and methods for compressing and speeding up dense matrix multiplications as found, for examples, in the fully connected and recurrent layers of neural networks for embedded large vocabulary continuous speech recognition (LVCSR). For compression, trace norm regularization technique embodiments were introduced and studied for training low rank factored versions of matrix multiplications. Compared to standard low rank training, the methods more consistently lead to good accuracy versus number of parameter trade-offs and can be used to speed-up training of large models. Faster inference may be further enabled on ARM processors through kernels optimized for small batch sizes, resulting in speed ups over the currently used library. Beyond LVCSR, the techniques are also generally applicable to embedded neural networks with large fully connected or recurrent layers.

Systems and methods for trace norm regularization and faster inference for embedded models

Described herein are systems and methods to identify and address sources of bias in an end-to-end speech model. In one or more embodiments, the end-to-end model may be a recurrent neural network with two 2D-convolutional input layers, followed by multiple bidirectional recurrent layers and one fully connected layer before a softmax layer. In one or more embodiments, the network is trained end-to-end using the CTC loss function to directly predict sequences of characters from log spectrograms of audio. With optimized recurrent layers and training together with alignment information, some unwanted bias induced by using purely forward only recurrences may be removed in a deployed model.

Systems and methods for principled bias reduction in production speech models

Described herein are systems and methods for creating and using Convolutional Recurrent Neural Networks (CRNNs) for small-footprint keyword spotting (KWS) systems. Inspired by the large-scale state-of-the-art speech recognition systems, in embodiments, the strengths of convolutional layers to utilize the structure in the data in time and frequency domains are combined with recurrent layers to utilize context for the entire processed frame. The effect of architecture parameters were examined to determine preferred model embodiments given the performance versus model size tradeoff. Various training strategies are provided to improve performance. In embodiments, using only &#x2dc;230 k parameters and yielding acceptably low latency, a CRNN model embodiment demonstrated high accuracy and robust performance in a wide range of environments.

Convolutional recurrent neural networks for small-footprint keyword spotting

Growing community of inventors

Santa Clara, CA, United States of America

Markus Kliegl