Microsoft Technology Licensing, LLC

Pengcheng He

Jianfeng Gao

Weizhu Chen

Xiaodong Liu

Zheng Chen

Yu Wang

Hoifung Poon

Matthew Brigham Hall

Yu Zhao

Wei Xiong

Yuting Sun

Hao Cheng

Yi-Min Wang

Nanshan Zeng

Eslam K Abdelreheem

Mingyu Wang

Jiayuan Huang

Junyan Chen

Katherine Winant Osborne

Song Wang

Jade Huang

Lesly Sadiht Miculicich Werlen

Yanling Xiong

Yuantao Wang

Yujia Xie

Vikas Bahirwani

Jiaye Xie

Liaoyuan Ning

Robert S Harrell

Systems and methods are provided for training and using a novel unified language foundation model. An encoder-decoder natural language model is obtained and various training data is obtained and used for training. The training process integrates a combination of replaced token detection, corrupted span reconstruction, and disentangled attention methodologies to produce a unified encoder-decoder model. The trained model is trained for performing both natural language understanding (NLU) tasks and natural language generation (NLG) tasks. Attention applied to the model is applied discretely to segmented chunks of encoded data during processing to improve the efficiency of applying attention by the model.

Pre-training a unified natural language model with corrupted span and replaced token detection

The disclosure herein describes training a text processing model to generate model output text data using input text data and a sentence count. A training data entry including input text data and output text data is obtained. A sentence count of the output text data is determined, and the output text data is labeled with a sentence count label and a sentence number label. Model output text data is generated with a text processing model using the input text data and determined sentence count as input data. Loss data associated with a difference between the generated model output text data and the labeled output text data is used to adjust the text processing model, which enables the model to be trained to produce output text data with a target sentence count in a computationally efficient manner.

Length-controlled text generation using a text processing model

This document relates to training of machine learning models. One example method involves providing a machine learning model having one or more mapping layers. The one or more mapping layers can include at least a first mapping layer configured to map components of pretraining examples into first representations in a space. The example method also includes performing a pretraining stage on the one or more mapping layers using the pretraining examples. The pretraining stage can include adding noise to the first representations of the components of the pretraining examples to obtain noise-adjusted first representations. The pretraining stage can also include performing a self-supervised learning process to pretrain the one or more mapping layers using at least the first representations of the training data items and the noise-adjusted first representations of the training data items.

Adversarial pretraining of machine learning models

This document relates to training of machine learning models such as neural networks. One example method involves providing a machine learning model having one or more layers and associated parameters and performing a pretraining stage on the parameters of the machine learning model to obtain pretrained parameters. The example method also involves performing a tuning stage on the machine learning model by using labeled training samples to tune the pretrained parameters. The tuning stage can include performing noise adjustment of the labeled training examples to obtain noise-adjusted training samples. The tuning stage can also include adjusting the pretrained parameters based at least on the labeled training examples and the noise-adjusted training examples to obtain adapted parameters. The example method can also include outputting a tuned machine learning model having the adapted parameters.

Adversarial training of machine learning models

A method for training a language model comprises (a) receiving vectorized training data as input to a multitask pretraining problem; (b) generating modified vectorized training data based on the vectorized training data, according to an upstream data embedding; (c) emitting pretraining output based on the modified vectorized training data, according to a downstream data embedding equivalent to the upstream data embedding; and (d) adjusting the upstream data embedding and the downstream data embedding by computing, based on the pretraining output, a gradient of the upstream data embedding disentangled from a gradient of the downstream data embedding, thereby advancing the multitask pretraining problem toward a pretrained state.

Language-model pretraining with gradient-disentangled embedding sharing

Systems and methods are provided for facilitating the building and use of natural language understanding models. The systems and methods identify a plurality of tokens and use them to generate one or more pre-trained natural language models using a transformer. The transformer disentangles the content embedding and positional embedding in the computation of its attention matrix. Systems and methods are also provided to facilitate self-training of the pre-trained natural language model by utilizing multi-step decoding to better reconstruct masked tokens and improve pre-training convergence.

Efficient transformer language models with disentangled attention and multi-step decoding

This document relates to architectures and training procedures for multi-task machine learning models, such as neural networks. One example method involves providing a multi-task machine learning model having one or more shared layers and two or more task-specific layers. The method can also involve performing a pretraining stage on the one or more shared layers using one or more unsupervised prediction tasks. The method can also involve performing a tuning stage on the one or more shared layers and the two or more task-specific layers using respective task-specific objectives.

Multi-task machine learning architectures and training procedures

Generally discussed herein are devices, systems, and methods for generating an embedding that is both local string dependent and global string dependent. The generated embedding can improve machine learning (ML) model performance. A method can include converting a string of words to a series of tokens, generating a local string-dependent embedding of each token of the series of tokens, generating a global string-dependent embedding of each token of the series of tokens, combining the local string dependent embedding the global string dependent embedding to generate an n-gram induced embedding of each token of the series of tokens, obtaining a masked language model (MLM) previously trained to generate a masked word prediction, and executing the MLM based on the n-based induced embedding of each token to generate the masked word prediction.

ML using n-gram induced input representation

Methods, systems, apparatuses, and computer program products are provided for extracting an entity value from a sentence. An embedding set that may include one or more sentence embeddings is generated for at least part of a first sentence that is tagged to associate a first named entity in the sentence with an entity type. A plurality of candidate embeddings is also generated for at least part of a second sentence. The one or more sentence embeddings in the embedding set may be compared with each of the plurality of candidate embeddings, and a match score may be assigned to each comparison to generate a match score set. A particular match score of the match score set may be identified that exceeds a similarity threshold, and an entity value of the entity type may be extracted from the second sentence associated with the identified match score.

Example based entity extraction, slot filling and value recommendation

A system for training and deploying an artificial conversational entity using an artificial intelligence (AI) based communications system is disclosed. The system may comprise a memory storing machine readable instructions. The system may also comprise a processor to execute the machine readable instructions to receive a request via an artificial conversational entity. The processor may also transmit a response to the request based on a dialog tree generated from at least a model-based action generator and a memory-based action generator. The processor may further provide a training option to a user in the event the response is suboptimal. The processor may additionally receive a selection from the user via the training option. The selection may be associated with an optimal response.

Interactive dialog training and communication system using artificial intelligence

Generally discussed herein are devices, systems, and methods for visualization of user intent in accessing a virtual agent. A method can include receiving sentences from the respective interaction sessions, projecting the sentences to a higher-dimensional space to create respective higher-dimensional vectors, projecting the higher-dimensional vectors to a lower-dimensional space to create respective lower-dimensional vectors, the lower-dimensional space including either two dimensions or three dimensions, plotting the lower-dimensional vectors as respective points on a graph, encoding the respective points consistent with the respective intents determined to be associated with the sentences by an intent classifier to create encoded points, and generating a visual representation of the encoded points.

Growing community of inventors

Sammamish, WA, United States of America

Pengcheng He