Microsoft Technology Licensing, LLC

Weizhu Chen

Jianfeng Gao

Pengcheng He

Xiaodong Liu

Yu Wang

Royi Ronen

Kunal Mukerjee

Jian-guang Lou

Hoifung Poon

Ankit Saraf

Woo Ram Lee

Eslam K Abdelreheem

Ganapathy V Raman

Erez Altus

Chao Li

Haiyuan Cao

Jiayuan Huang

Yi Mao

Yu Hu

Abedelkader Asi

Aarohi Arora

Yarin Kuper

Manoj Ajith Prasad

PalaniRaj Kaliyaperumal

Kuralmani Elango

Sparky Toews

Santosh Vasant Bhosale

Aritra DattaGupta

Roger F Johnson

Hao Cheng

Vladyslav Bazyliak

Song Wang

Somak Mitra

Bryan Wilhelm

Benjamin Reich

Matthew Justice

Will Qian

Changhong Yuan

Zeqi Lin

Yi Liu

Olga Goldenberg

Shimrit Rada Bemis

Xiaoying Guo

Yunjing Ma

Shean Wang

Tsuyoshi Watanabe

Jaclyn Ruth Elizabeth Phillips

Yuantao Wang

Justin Carl Wong

Nan Zhao

Michael J Nelson

The disclosure herein describes reducing training bias in outputs generated by a generative language model. A communication segment associated with a communication is obtained by at least one processor of a generative language model. An output value associated with the communication segment is generated by the generative language model. The output value is mapped to a set of training bias values associated with the generative language model and based on the mapping of the output value to a training bias value of the set of training bias values, an alternative output value is generated. The alternative output value is used in a generated segment output for the communication segment. The accuracy of segment outputs generated by the generative language model is improved through reducing or eliminating its training biases.

Reducing biases of generative language models

In some examples, iterative sampling based dataset clustering may include sampling a dataset that includes a plurality of items to identify a specified number of sampled items. The sampled items may be clustered to generate a plurality of clusters. Un-sampled items may be assigned from the plurality of items to the clusters. Remaining un-sampled items that are not assigned to the clusters may be identified. A ratio associated with the remaining un-sampled items and the plurality of items may be compared to a specified threshold. Based on a determination that the ratio is greater than the specified threshold, an indication of completion of clustering of the plurality of items may be generated.

Iterative sampling based dataset clustering

This document relates to training of machine learning models. One example method involves providing a machine learning model having one or more mapping layers. The one or more mapping layers can include at least a first mapping layer configured to map components of pretraining examples into first representations in a space. The example method also includes performing a pretraining stage on the one or more mapping layers using the pretraining examples. The pretraining stage can include adding noise to the first representations of the components of the pretraining examples to obtain noise-adjusted first representations. The pretraining stage can also include performing a self-supervised learning process to pretrain the one or more mapping layers using at least the first representations of the training data items and the noise-adjusted first representations of the training data items.

Adversarial pretraining of machine learning models

This document relates to training of machine learning models such as neural networks. One example method involves providing a machine learning model having one or more layers and associated parameters and performing a pretraining stage on the parameters of the machine learning model to obtain pretrained parameters. The example method also involves performing a tuning stage on the machine learning model by using labeled training samples to tune the pretrained parameters. The tuning stage can include performing noise adjustment of the labeled training examples to obtain noise-adjusted training samples. The tuning stage can also include adjusting the pretrained parameters based at least on the labeled training examples and the noise-adjusted training examples to obtain adapted parameters. The example method can also include outputting a tuned machine learning model having the adapted parameters.

Adversarial training of machine learning models

A method for training a language model comprises (a) receiving vectorized training data as input to a multitask pretraining problem; (b) generating modified vectorized training data based on the vectorized training data, according to an upstream data embedding; (c) emitting pretraining output based on the modified vectorized training data, according to a downstream data embedding equivalent to the upstream data embedding; and (d) adjusting the upstream data embedding and the downstream data embedding by computing, based on the pretraining output, a gradient of the upstream data embedding disentangled from a gradient of the downstream data embedding, thereby advancing the multitask pretraining problem toward a pretrained state.

Language-model pretraining with gradient-disentangled embedding sharing

Examples described herein generate training data for machine learning (ML) for natural language (NL) processing (such as semantic parsing for translating NL). A formula tree is generated based on sampling both a formula grammar and NL templates. Using the formula tree, an ML training data instance pair is generated comprising a formula example and an NL example. A context example may also be used during instantiation of the formula tree. An ML model is trained with training data including the ML training data instance pair, and ML output is generated from NL input. The ML output includes, for example, a machine-interpretable formula, a database querying language command, or a general programming language instruction. Some examples support context-free grammar, probabilistic context-free grammar, and/or non-context-free production rules.

Generating model training data from a domain specification

Systems and methods are provided for facilitating the building and use of natural language understanding models. The systems and methods identify a plurality of tokens and use them to generate one or more pre-trained natural language models using a transformer. The transformer disentangles the content embedding and positional embedding in the computation of its attention matrix. Systems and methods are also provided to facilitate self-training of the pre-trained natural language model by utilizing multi-step decoding to better reconstruct masked tokens and improve pre-training convergence.

Efficient transformer language models with disentangled attention and multi-step decoding

This document relates to architectures and training procedures for multi-task machine learning models, such as neural networks. One example method involves providing a multi-task machine learning model having one or more shared layers and two or more task-specific layers. The method can also involve performing a pretraining stage on the one or more shared layers using one or more unsupervised prediction tasks. The method can also involve performing a tuning stage on the one or more shared layers and the two or more task-specific layers using respective task-specific objectives.

Multi-task machine learning architectures and training procedures

Growing community of inventors

Kirkland, WA, United States of America

Weizhu Chen