Microsoft Technology Licensing, LLC

Vu Minh Le

Sumit Gulwani

Gustavo Araujo Soares

Mark Alistair Wilson-Thomas

Arjun Radhakrishna

Ashish Tiwari

David Ellis Pugh

Ivan Radicek

José Pablo Cambronero Sánchez

Carina Suzana Negreanu

Piyush Arora

Titus Barik

Wode Ni

Ameya Sanjay Ketkar

Gust Ben Anneloes Verbruggen

Nikolai Tillmann

Benjamin Zorn

Andrew Donald Gordon

Daniel Morris

Fang Liu

Abhishek Udupa

Rishabh Singh

Jonathan Paul De Halleux

Vivek Gupta

Advait Sarkar

Daniel Galen Simmons

Avrilia Floratou

Mark Marron

Ranvijay Kumar

Oleksandr Polozov

Peter Groenewegen

Chiu Ying Cheung

Irene Rogan Shaffer

Ehimwenma Nosakhare

Joyce Yu Cahoon

Jordan Joseph Henkel

Brian Paul Slininger

Wee Hyong Tok

Jason Wang

Mukul Singh

Shaily Jignesh Fozdar

Jialu Zhang

Abhishek Narain

Gust Verbruggen

Noelle Yanhui Li

David Joseph Donahue

Daniel Muñoz Huerta

Richard Anarfi

Christian Leopold Bejamin Poelitz

Kiarash Rahmani

Yan Qiu

Rohan Jayesh Bavishi

Anna Fariha

Mohammad Raza

Anders Miltner

Mikael Mayer

Maxim Grechkin

Zhendong Su

The indirect querying of models to determine capabilities possessed by the model. Such indirect queries take the form of model input that potentially includes a natural language input user data. Such model input is structured such that the output of the model is either not natural language at all, or else is natural language that is not semantically responsive to the natural language input. Nevertheless, the output is evaluated to estimate or determine the capability possessed by the model. Thus, models may be more fully utilized to their better potential.

Model capability extraction

A computer-implemented method for performing natural language-based data integration includes causing execution of a data integration application on a remote device via a network and causing surfacing of a GUI corresponding to the data integration application on a display of the remote device. The method includes receiving, via the GUI, a natural language input representing a data integration task, generating, via an LLM, a set of ordered activities corresponding to the data integration task represented by the natural language input, and selecting, via the LLM, one or more APIs for performing each activity within the set of ordered activities. The method also includes generating a data pipeline based on the set of ordered activities and the API(s) for performing each activity, as well as back-translating the data pipeline to a desired data format for execution by the data integration application.

Natural language-based data integration

Some embodiments generate prompts and submit them in queries to a language model trained on code to perform automated program repair. Some embodiments fix syntactic mistakes and semantic mistakes by combining multimodal prompts, iterative querying, test-case-based selection of few-shots, and program chunking. In some cases, edit distance is minimized between an initial flawed program and the automatically created improved version of that program. The initial flawed program is obtained from a programming student, or from a source code generator.

Program improvement using large language models

Techniques are described herein that are capable of creating a language-agnostic computer program repair engine generator. A context-free grammar is annotated to identify token(s) that are likely to be included in or excluded from a computer program in a manner that violates the context-free grammar. A language-agnostic computer program repair engine generator is created that is configured to generate a parser. The repair engine generator is configured to create a repair engine that: converts the candidate string into repaired strings that neither violate the context-free grammar nor violate a criterion for a valid computer program; calculates differences between the candidate string and the respective repaired strings; and replaces the candidate string with a designated repaired string based at least in part on the difference between the designated repaired string and the candidate string being less than or equal to a difference threshold.

Language-agnostic computer program repair engine generator

Some embodiments perform fully unsupervised string error detection and repair, for strings containing both syntactic errors and semantic errors. In some embodiments, string error mitigation includes learning a regular-expression-based pattern that characterizes at least a predefined fraction of a set of string data values, identifying and masking a semantic substring, deriving edits to repair a string which contains the semantic substring, and replacing a mask value with a replacement substring that conforms to the pattern. Some embodiments utilize a language model. Some embodiments include or utilize a pattern learner, one or more learned significant patterns, and a repair engine. Some embodiments exploit program execution information to generate patterns when significant patterns are not initially found.

String data error detection and repair

Edit automation functionality generalizes edits performed by a user in a document, locates similar text, and recommends or applies transforms while staying within a current workflow. Source code edits such as refactoring are automated. The functionality uses or provides anchor target lists, temporal edit patterns, edit graphs, automatable edit sequence libraries, and other data structures and computational techniques for identifying locations appropriate for particular edits, for getting transforms, for selecting optimal transforms, for leveraging transforms in an editing session or later, and for displaying transform recommendations and results. The edit automation functionality enhances automation subtool generation, discoverability, and flexibility, for refactoring, snippet insertion, quick actions in an integrated development environment, and other automatable edit sequences.

Edit automation using an anchor target list

Embodiments use a multi-modal approach to generate software programs that match a solution program description. The solution program description may include natural language, input-output examples, partial source code, desired operators, or other hints. Some embodiments use optimized prompts to a pre-trained language model to obtain initial candidate programs. Maximal program components are extracted and then recombined variously using component-based synthesis. Beam search reduces a solution program search space by discarding some candidates from a given synthesis iteration. Relevance metrics, string similarity metrics, operator frequency distributions, token rareness scores, and other optimizations may be employed. By virtue of optimizations and the multi-modal approach, a solution program may be obtained after fewer iterations than by use of a language model alone. The multi-modal approach is domain agnostic, as illustrated by examples using regular expression and cascading style sheet selector domain specific languages.

Multi-modal program inference

Edit automation using a temporal edit pattern

Distinguishing pattern differences from non-pattern differences. A set of differences is identified. The set comprises a plurality of differences between first and second versions of a document. A pattern is identified. The pattern explains a transformation from a first string in the first version of the document to a second string in the second version of the document. A subset of differences are identified. The subset comprises a plurality of differences, from among the set, which match the pattern. While presenting a user interface that visually highlights differences between the first and second versions of the document, a first visual treatment is applied to a first difference, based on the first difference being included in the subset. A second visual treatment is also applied to a second difference, based on the second difference being excluded from the subset. The second visual treatment is different than the first visual treatment.

Distinguishing pattern differences from non-pattern differences

Described herein are various technologies pertaining to identifying boundaries of a substring in a log file, wherein the substring is to be extracted from the log file and used to construct a table. An indication is received that a user has selected a beginning boundary of the substring, and the ending boundary of the substring is automatically identified.

Identifying boundaries of substrings to be extracted from log files

Flexible yet efficient 'find' operations search source code for matches to a general pattern after a developer provides an example string that matches the pattern, without requiring the developer to write a regular expression or script that will implement the desired pattern. Example-driven find-replace functionality uses regular expressions or other pattern match codes, and scripts or other transforms, which are synthesized automatically from examples provided by a developer. This technology allows the developer to focus on workflow inside an integrated development environment instead of breaking focus to search for external documentation, or unfortunately foregoing the flexibility and power of regular expressions and scripts. Synthesizer outputs may be directly or indirectly ranked through user feedback, allowing their refinement. Find match generality may be controlled, e.g., by specifying regex star positions or star counts. Entry of guiding examples may be assisted by autocompletion. Performance criteria are also described.

Source code text replacement by example

Automatically identifying context-specific repeated transformations (such as repeated edit tasks) that are based on observation of the developer drafting or modifying code. As the developer modifies the code, the code passes through a series of states, one after the other. The computing system observes the series of states of the code. It is based on this observation that the computing system identifies repeated transformations of the code for potentially offering to continue performing the repeated transformations for the user. This alleviates the developer from having to manually perform the remainder of the repeated transformations.

Automatic repetition of context-specific code edits

Systems and techniques for structured query language (SQL) query formatting by examples are described herein. In an example, a SQL query formatter is adapted to receive a SQL statement, such that the SQL statement is formed using SQL tokens. The SQL query formatter may be further adapted to use a SQL formatter model to format the SQL statement, which results in a formatted SQL statement. The SQL formatter model may be trained using at least one previously executed SQL statement. The SQL formatter model may include format definitions for SQL tokens. The SQL query formatter may be further adapted to output the formatted SQL statement.

SQL query formatting by examples

Described herein are various technologies pertaining to processing a log file, such that the processed log file is well-suited for analysis by a programming by example (PBE) system. When processing the log file, header lines and comment lines in the log file can be automatically identified and filtered from the log file, such that the PBE system does not contemplate the header lines and comment lines. Further, when processing the log file, boundaries of records in the log file can be identified without requiring user input. The PBE system constructs a table based upon the identified boundaries of the records in the log file, wherein the table includes rows that correspond to the records, and further wherein the table fails to include rows that correspond to header lines and comment lines.

Identifying header lines and comment lines in log files

Various technologies pertaining to extracting data encoded in a tree-structured document and generating a table based upon the extracted data are described herein. In a first embodiment, the table is generated without requiring input from a data cleaner. In a second embodiment, the table is generated based upon examples set forth by a data cleaner.

Generating tables based upon data extracted from tree-structured documents

Growing community of inventors

Kirkland, WA, United States of America

Vu Minh Le