Baidu USA LLC

Mingbo Ma

Renjie Zheng

Liang Huang

Haifeng Wang

Hua Wu

Zhongjun He

Hao Xiong

Chuanqiang Zhang

Xing Li

Hairong Liu

Kaibo Liu

Baigong Zheng

Junkun Chen

Representation learning for text and speech has improved many language-related tasks. However, existing methods only learn from one input modality, while a unified representation for both speech and text is needed for tasks such as end-to-end speech translation. Consequently, these methods cannot exploit various large-scale text and speech data and their performance is limited by the scarcity of parallel speech translation data. To address these problems, embodiments of a fused acoustic and text masked language model (FAT-MLM) are disclosed. FAT-MLM embodiments jointly learn a unified representation for both acoustic and text input from various types of corpora including parallel data for speech recognition and machine translation, and pure speech and text data. Within this cross-modal representation learning framework, an end-to-end model is further presented for fused acoustic and text speech translation. Experiments show that by fine-tuning from FAT-MLM, the speech translation model embodiments substantially improve translation quality.

Fused acoustic and text encoding for multimodal bilingual pretraining and speech translation

Presented herein are embodiments of a prefix-to-prefix framework for simultaneous translation that implicitly learns to anticipates in a single translation. Within these frameworks are effective 'wait-k' policy model embodiments that may be trained to generate a target sentence concurrently with a source sentence but lag behind by a predefined number of words. Embodiments of the prefix-to-prefix framework achieve low latency and better quality when compared to full-sentence translation in four directions: Chinese&#x2194;English and German&#x2194;English. Also presented herein is a novel latency metric that addresses deficiencies of previous latency metrics.

Systems and methods for simultaneous translation with integrated anticipation and controllable latency (STACL)

Growing community of inventors

Milpitas, CA, United States of America

Mingbo Ma