Microsoft Technology Licensing, LLC

Nan Duan

Ming Zhou

Jian Jiao

Yeyun Gong

Neelakantan Sundaresan

Alexey Svyatkovskiy

Shuai Lu

Shengyu Fu

Ruofei Zhang

Lei Ji

Eren Manavoglu

Duyu Tang

Xingwei He

Yaobo Liang

Jiun-Hung Chen

Pengcheng Wang

Ying Qiao

Systems and methods are provided for generating a keyword sequence from an input query. A first text sequence corresponding to an input query may be received and encoded into a source sequence representation using an encoder of a machine learning model. A keyword sentence may then be generated from the source sequence representation using a decoder of the machine learning model. The decoder may generate a modified generation score for a plurality of prediction tokens, wherein the modified generation score is based on the respective prediction token generation score and a maximum generation score for a suffix of each prediction token. The decoder may then select the prediction token of the plurality of prediction tokens based on the modified generation score, and add the selected prediction token to the previously decoded partial hypothesis provided by the decoder.

Look ahead strategy for trie-based beam search in generative retrieval

A large language model, previously pre-trained on multiple source code modeling tasks, is pre-trained, through curriculum learning, to learn to predict a code execution trace given a source code program. The model is pre-trained using a variety of pre-training datasets consisting of pairs of a source code sample and a corresponding execution trace. The curriculum pre-training starts with a pre-training dataset of single line executions and adds in additional pre-training datasets with more increasing complex behaviors. The pre-training datasets include mutation-augmented source code samples and their corresponding execution traces.

Code execution trace generation with pre-trained large language model

A deep learning model is pre-trained with a large-scale of unsupervised data of code review tasks in order to learn the relationships between code changes and a code review. The pre-trained deep learning model predicts a code review given a code diff hunk in a code diff format. The code diff hunk includes the changed code and its surrounding context. The pre-trained deep learning model may then be fine-tuned with supervised data in order to make predictions for several code review activities, such as, code change quality estimation and code refinement.

Pre-training for automating code review activities

A transformer-based neural network includes at least one mask attention network (MAN). The MAN computes an original attention data structure that expresses influence between pairs of data items in a sequence of data items. The MAN then modifies the original data structure by mask values in a mask data structure, to produce a modified attention data structure. Compared to the original attention data structure, the modified attention data structure better accounts for the influence of neighboring data items in the sequence of data items, given a particular data item under consideration. The mask data structure used by the MAN can have static and/or machine-trained mask values. In one implementation, the transformer-based neural network includes at least one MAN in combination with at least one other attention network that does not use a mask data structure, and at least one feed-forward neural network.

Transformer-based neural network including a mask attention network

The present disclosure provides a technical solution of multi-modal chatting, which may provide response to user query by using multi-modal response in the interaction between chatbot and human beings, so that the expressing ways and the expressed content by the chatbot could be richer by using such response in a multi-modal way.

Videochat

Technologies described herein relate to dense retrieval and ranking of search results. A query indicating a computing context or user input is received. An embedding of the query is computed by way of a first encoder, and candidate results selected from a pool of potential results based upon the embedding of the query and embeddings of the potential results. A similarity score for a first of the candidate results is computed by way of a second encoder trained based upon an order metric that defines a ranking over a training set of potential results. The first encoder is trained based upon output of the second encoder prior to computing the embedding of the query. The candidate results are ranked based upon the similarity score of the first candidate result, and results responsive to the query are identified based upon the ranking. The identified results are output to a computing device.

Dense retrieval employing progressive distillation training

Implementations of the present disclosure relate to text generation with a customizable style. In a method, a first natural language is received; the first natural language text is converted, via a text generation model, into a second natural language text that at least partly reflects the meaning of the first natural language text and has a style distinguishable from the first natural language text, the text generation model comprising a modifiable parameter; and in response to receiving a modification to the parameter, the first natural language text is converted, via the text generation model, into a third natural language text that at least partly reflects the meaning of the first natural language text and includes a style distinguishable from both the first natural language text and the second natural language text.

Text generation with customizable style

A retrieval-augmented code completion system uses the context of a partially-formed source code snippet of a source code program and a hint to predict the source code tokens needed to complete the partially-formed source code snippet. The hint is a source code segment that completes a semantically-similar source code segment of the partially-formed source code snippet. The hint is found in a retrieval source code database using a hybrid retrieval technique. A deep learning decoder model uses the context of the partially-formed source code snippet and the hint to predict the most likely candidate sequence of source code tokens to complete the partially-formed source code snippet.

Retrieval-augmented code completion

A training system produces a resource-efficient machine-trained model via a training architecture that employs plural processing paths. Some of the processing paths incorporate the use of auxiliary information that imparts external knowledge about source items being processed. The training architecture also employs contrastive learning that operates at different respective levels within the training architecture. For instance, the training architecture uses encoder-level contrastive learning to compare output information generated by different encoders within the training architecture. The training architecture uses decoder-level contrastive learning to compare output information produced by different decoders within the training architecture. An inference-stage system performs an application task using the model produced by the training system.

Resource-efficient sequence generation with dual-level contrastive learning

In embodiments of the present disclosure, there is provided an approach for fact checking based on semantic graphs. According to embodiments of the present disclosure, after obtaining a text to be fact checked, a plurality of evidence sentences related to the text are retrieved from an evidence database. Then, semantic graphs of the text and the evidence sentences are constructed based on the semantic analysis, and a veracity of a statement in the text can be determined based on the semantic graphs. Embodiments of the present disclosure propose a graph-based reasoning approach for fact checking, and use the constructed semantic graphs to facilitate verification of the truthfulness of the text, thereby improving the accuracy for fact checking.

Growing community of inventors

Beijing, China

Nan Duan