Microsoft Technology Licensing, LLC

Alexey Svyatkovskiy

Neelakantan Sundaresan

Colin Bruce Clement

Michele Tufano

Shao Kun Deng

Shengyu Fu

Dawn Drain

Shuai Lu

Miltiadis Allamanis

Xiaoyu Liu

Nan Duan

Daya Guo

Ying Zhao

Duyu Tang

Negar Ghorbani

Andrei Zlotchevski

Mark Alistair Wilson-Thomas

Roshanak Zilouchian Moghaddam

David Ellis Pugh

Todd Douglas Mytkowicz

Shuvendu K Lahiri

Christian Alma Bird

German David Obando Chacon

Yevhen Mohylevskyy

Shuo Zhang

Chen Wu

Peter Groenewegen

Jui Hanamshet

David Alberto Nader Palacio

Christian Bird

James Drain

An Artificial Intelligence (AI) driven pull request summarization system utilizes a large language model to classify the code changes of a pull request and to generate a summary of the changes contained in the pull request. The large language model predicts the pull request classification and summarization given a prompt that includes the top-k code changes in the pull request of a repository. The code changes are prioritized based on the most impact that a code change has on the files, methods and classes in the repository. Each of the top-k code changes is linked to a related open issue of the repository, if any. A suggested code reviewer for a code change is then selected from an author or commentator associated with the linked open issue.

AI-driven pull request summarization

A code completion system for a CLI utilizes neural transformer models with attention to generate candidates to complete a line of CLI code. The code completion system uses a first deep learning model to predict at most k candidate command names to follow n immediately preceding lines of CLI code which are presented to a developer. Upon the developer accepting one of the candidate command names, the code completion system uses a second deep learning model to predict at most k parameter strings to complete the line of CLI code.

Neural transformer code completion for command line interface

A code completion tool uses a neural transformer model with attention to generate candidate sequences to complete a method body of a method signature. The neural transformer model is trained with source code programs and natural language text. The neural transformer model learns the meaning of a method name, its corresponding method parameters and types from a large corpus of unsupervised dataset of source code methods and a supervised dataset of tasks including source code constructs in combination with natural language docstrings to infer a candidate sequence of subtokens that represent a method body for a particular method signature.

Neural method completion based on natural language and source code

An automated program repair tool utilizes a neural transformer model with attention to predict the contents of a bug repair in the context of source code having a bug of an identified bug type. The neural transformer model is trained on a large unsupervised corpus of source code using a span-masking denoising optimization objective, and fine-tuned on a large supervised dataset of triplets containing a bug-type annotation, software bug, and repair. The bug-type annotation is derived from an interprocedural static code analyzer. A bug type edit centroid is computed for each bug type and used in the inference decoding phase to generate the bug repair.

Automated program repair tool

A large language model, previously pre-trained on multiple source code modeling tasks, is pre-trained, through curriculum learning, to learn to predict a code execution trace given a source code program. The model is pre-trained using a variety of pre-training datasets consisting of pairs of a source code sample and a corresponding execution trace. The curriculum pre-training starts with a pre-training dataset of single line executions and adds in additional pre-training datasets with more increasing complex behaviors. The pre-training datasets include mutation-augmented source code samples and their corresponding execution traces.

Code execution trace generation with pre-trained large language model

A deep learning model is pre-trained with a large-scale of unsupervised data of code review tasks in order to learn the relationships between code changes and a code review. The pre-trained deep learning model predicts a code review given a code diff hunk in a code diff format. The code diff hunk includes the changed code and its surrounding context. The pre-trained deep learning model may then be fine-tuned with supervised data in order to make predictions for several code review activities, such as, code change quality estimation and code refinement.

Pre-training for automating code review activities

A transfer learning system is used for the development of neural transformer models pertaining to software engineering tasks. The transfer learning system trains source code domain neural transformer models with attention in various configurations on a large corpus of unsupervised training dataset of source code programs and/or source code-related natural language text. A web service provides the trained models for use in developing a model that may be fine-tuned on a supervised training dataset associated with a software engineering task thereby generating a tool to perform the software engineering task.

Transfer learning system for automated software engineering tasks

A deep learning model trained to learn to predict source code is tuned for a target source code generation task through reinforcement learning using a reward score that considers the quality of the source code predicted during the tuning process. The reward score is adjusted to consider code-quality factors and source code metrics. The code-quality factors account for the predicted source code having syntactic correctness, successful compilation, successful execution, successful invocation, readability, functional correctness, and coverage. The source code metrics generate a score based on how close the predicted source code is to a ground truth code.

Code generation through reinforcement learning using code-quality rewards

Custom source code generation models are generated by tuning a pre-trained deep learning model by freezing the model parameters and optimizing a prefix. The tuning process is distributed across a user space and a model space where the embedding and output layers are performed in the user space and the execution of the model is performed in a model space that is isolated from the user space. The tuning process updates the embeddings of the prefix across the separate execution spaces in a manner that preserves the privacy of the data used in the tuning process.

Custom models for source code generation via prefix-tuning

A code completion tool uses a neural transformer model to generate candidate sequences to complete a line of source code. The neural transformer model is trained using a conditional language modeling objective on a large unsupervised dataset that includes source code programs written in several different programming languages. The neural transformer model is used within a beam search that predicts the most likely candidate sequences for a code snippet under development.

Multi-lingual line-of-code completion system

A code generation system uses a non-terminal expansion model and a non-terminal selector model to generate a code sketch to complete a partially-formed source code snippet. The non-terminal expansion model is a neural transformer model trained on a supervised dataset through reinforcement learning to learn to predict the production rule to expand for a given non-terminal symbol. The non-terminal selector model is trained through reinforcement learning to predict the non-terminal symbol to expand given a partial-code state. The models are used in a two-step beam search to generate the top candidate code sketches, where a candidate code sketch may contain a hole that represents an unexpanded non-terminal symbol.

Code generation with reinforcement learning

A code insertion engine predicts one or more statements of a programming language to be inserted at an insertion point in between existing source code statements of a source code program being edited. The code insertion engine extracts the surrounding context of the insertion point which includes the source code immediately preceding and the source code immediately following the insertion point. The code insertion engine uses a neural expansion model and a neural selector model to predict the one or more statements most likely to be inserted into the insertion point that are syntactically and semantically consistent with the surrounding context of the existing program.

Code insertion completion

A unit test generation system employs a neural transformer model with attention to generate candidate unit test sequences given a focal method of a programming language. The neural transformer model is pre-trained with source code programs and natural language text and fine-tuned with mapped test case pairs. A mapped test case pair includes a focal method and a unit test case for the focal method. In this manner, the neural transformer model is trained to learn the semantics and statistical properties of a natural language, the syntax of a programming language and the relationships between the code elements of the programming language and the syntax of a unit test case.

Unit test case generation with transformers

A retrieval-augmented neural transformer model with chunk cross-attention predicts a code review given a proposed source code change, represented as a code diff hunk, and a set of historical code review comments. The code diff hunk represents proposed edits to a source code snippet with its surrounding context that has not been changed. The historical code review comments are associated with code edits that are semantically similar to the proposed source code changes. The code diff hunk is partitioned into chunks which are used to find semantically similar historical code review comments. The set of historical code review comments is aggregated and used to guide the model in makings its predictions.

Code review comment generation via retrieval-augmented transformer with chunk cross-attention

An automated system for resolving program merges uses a multi-task neural transformer with attention. Each component of a merge conflict tuple (A, B, O) is represented as an AST and transformed into aligned AST-node sequences and aligned editing sequences. The multi-task neural transformer model predicts the tree editing steps needed to resolve the merge conflict and applies them to the AST representation of the code base. The tree editing steps include the edit actions that needed to be applied to the AST of the code base and the edit labels that are inserted or updated with the edit actions.

Growing community of inventors

Bellevue, WA, United States of America

Alexey Svyatkovskiy