Microsoft Technology Licensing, LLC

Microsoft Corporation

Zhigu Holdings Limited

Xuedong David Huang

Furu Wei

Shujie Liu

Chengyi Wang

Kenichi Kumatani

Zhengyou Zhang

Alejandro Acero

Li Deng

Michael Jack Sinclair

Yao Qian

Zicheng Liu

Jianfeng Gao

Yong Rui

Nanshan Zeng

Yu Wu

James G Droppo

Pengcheng He

Baolin Peng

Wei Xiong

Samuel Y Shen

Zheng Chen

Peng Bai

Xiaochuan Ni

Zhimin Zhang

Jian-Tao Sun

Hongjiang Zhang

Yanli Zheng

Systems and methods are provided for training and using a novel unified language foundation model. An encoder-decoder natural language model is obtained and various training data is obtained and used for training. The training process integrates a combination of replaced token detection, corrupted span reconstruction, and disentangled attention methodologies to produce a unified encoder-decoder model. The trained model is trained for performing both natural language understanding (NLU) tasks and natural language generation (NLG) tasks. Attention applied to the model is applied discretely to segmented chunks of encoded data during processing to improve the efficiency of applying attention by the model.

Pre-training a unified natural language model with corrupted span and replaced token detection

A system obtains a first training data set comprising labeled speech data or both labeled and unlabeled data corresponding to a high-resource data set as well as latent speech representations based on the first training data set. The system trains a machine learning model on the first training data set to learn phonetically aware speech representations corresponding to the first training data set. The system applies the latent speech representations to a transformer context network to generate contextual representations. The system aligns each of the contextual representations with a phoneme label to generate phonetically-aware contextual representations. The system causes a refinement engine to further refine the machine learning model.

Unified speech representation learning

Systems and methods are provided for training a machine learning model to learn speech representations. Labeled speech data or both labeled and unlabeled data sets is applied to a feature extractor of a machine learning model to generate latent speech representations. The latent speech representations are applied to a quantizer to generate quantized latent speech representations and to a transformer context network to generate contextual representations. Each contextual representation included in the contextual representations is aligned with a phoneme label to generate phonetically-aware contextual representations. Quantized latent representations are aligned with phoneme labels to generate phonetically aware latent speech representations. Systems and methods also include randomly replacing a sub-set of the contextual representations with quantized latent speech representations during their alignments to phoneme labels and aligning the phonetically aware latent speech representations to the contextual representations using supervised learning.

Growing community of inventors

Yarrow Point, WA, United States of America

Xuedong David Huang