Baidu USA LLC

Hairong Liu

Sanjeev Satheesh

Zhenyao Zhu

Haifeng Wang

Hua Wu

Zhongjun He

Sercan Omer Arik

Gregory Diamos

Hao Xiong

Adam Coates

Chuanqiang Zhang

Xing Li

Haonan Yu

Eric Dean Battenberg

Yanqi Zhou

Christopher Fougner

Anuroop Sriram

Heewoo Jun

Markus Kliegl

Vinay Rao

Yashesh Gaur

Mingbo Ma

Ajay Kannan

Liang Huang

Rewon Child

Renjie Zheng

Kaibo Liu

Jiaji Huang

Atul Kumar

David Seetapun

Baigong Zheng

Siavash Ebrahimi

Neural Architecture Search (NAS) is a laborious process. Prior work on automated NAS targets mainly on improving accuracy but lacked consideration of computational resource use. Presented herein are embodiments of a Resource-Efficient Neural Architect (RENA), an efficient resource-constrained NAS using reinforcement learning with network embedding. RENA embodiments use a policy network to process the network embeddings to generate new configurations. Example demonstrates of RENA embodiments on image recognition and keyword spotting (KWS) problems are also presented herein. RENA embodiments can find novel architectures that achieve high performance even with tight resource constraints. For the CIFAR10 dataset, the tested embodiment achieved 2.95% test error when compute intensity is greater than 100 FLOPs/byte, and 3.87% test error when model size was less than 3M parameters. For the Google Speech Commands Dataset, the tested RENA embodiment achieved the state-of-the-art accuracy without resource constraints, and it outperformed the optimized architectures with tight resource constraints.

Resource-efficient neural architects

Presented herein are embodiments of a prefix-to-prefix framework for simultaneous translation that implicitly learns to anticipates in a single translation. Within these frameworks are effective 'wait-k' policy model embodiments that may be trained to generate a target sentence concurrently with a source sentence but lag behind by a predefined number of words. Embodiments of the prefix-to-prefix framework achieve low latency and better quality when compared to full-sentence translation in four directions: Chinese&#x2194;English and German&#x2194;English. Also presented herein is a novel latency metric that addresses deficiencies of previous latency metrics.

Systems and methods for simultaneous translation with integrated anticipation and controllable latency (STACL)

Described herein are systems and methods to identify and address sources of bias in an end-to-end speech model. In one or more embodiments, the end-to-end model may be a recurrent neural network with two 2D-convolutional input layers, followed by multiple bidirectional recurrent layers and one fully connected layer before a softmax layer. In one or more embodiments, the network is trained end-to-end using the CTC loss function to directly predict sequences of characters from log spectrograms of audio. With optimized recurrent layers and training together with alignment information, some unwanted bias induced by using purely forward only recurrences may be removed in a deployed model.

Systems and methods for principled bias reduction in production speech models

Described herein are systems and methods for automatic unit selection and target decomposition for sequence labelling. Embodiments include a new loss function called Gram-Connectionist Temporal Classification (CTC) loss that extend the popular CTC loss function criterion to alleviate prior limitations. While preserving the advantages of CTC, Gram-CTC automatically learns the best set of basic units (grams), as well as the most suitable decomposition of target sequences. Unlike CTC, embodiments of Gram-CTC allow a model to output variable number of characters at each time step, which enables the model to capture longer term dependency and improves the computational efficiency. It is also demonstrated that embodiments of Gram-CTC improve CTC in terms of both performance and efficiency on the large vocabulary speech recognition task at multiple scales of data, and that systems that employ an embodiment of Gram-CTC can outperform the state-of-the-art on a standard speech benchmark.

Systems and methods for automatic unit selection and target decomposition for sequence labelling

Growing community of inventors

San Jose, CA, United States of America

Hairong Liu