Microsoft Technology Licensing, LLC

Google Inc.

Dengyong Zhou

Christopher J C Burges

Tao Tao

Quoc V Le

Yang Song

Raghav Gupta

Sanqiang Zhao

John Carlton Platt

Robert L Rounthwaite

Xian-Sheng Hua

Pushmeet Kohli

Kun Zhang

Sumit Basu

Yingwei Cui

Christopher John Brockett

Andrew M Dai

Wei Yu

Rishabh Singh

Christopher J Burges

Lihong Li

Zhong Wu

Bo Geng

Ed Huai-Hsin Chi

Emilio Parisotto

David J Bieber

Nan Du

Augustus Quadrozzi Odena

Yi Mao

Aakanksha Chowdhery

Jacob Austin

Maarten Paul Bosma

David Martin Dohan

Abdelrahman S A Mohamed

Dale Eric Schuurmans

Jason Weng Wei

Chen Liang

Olivier Jean Andrè Bousquet

Le Hou

Charles Aloysius Sutton

Guy Gur-Ari Krakover

Jiageng Luan

Anders Johan Andreassen

Xuezhi Wang

Yuan Xue

Xinyun Chen

Zhen Xu

Henryk Michalewski

Aitor Lewkowycz

Nathanael Schärli

Maxwell Nye

Sharan Narang

Nathan Scales

Stanley Kok

An example technique for image analysis is provided. An example image analysis method includes obtaining an instructive sequence descriptive of an instructive query, an instructive response, and an instructive trace of intermediate states from the instructive query to the instructive response. The example image analysis method includes inputting, to a machine-learned model, the instructive sequence and an operative image processing query that comprises image data, wherein the machine-learned model is configured to process the operative query with attention over the instructive sequence. The example method can include generating, using the machine-learned model and responsive to the operative query, an operative image processing response that comprises an analysis of the image data.

Image analysis by prompting of machine-learned models using chain of thought

Provided is a knowledge distillation technique for training a student language model that, relative to a larger teacher language model, has a significantly smaller vocabulary, lower embedding dimensions, and/or hidden state dimensions. Specifically, aspects of the present disclosure are directed to a dual-training mechanism that trains the teacher and student language models simultaneously to obtain optimal word embeddings for the student vocabulary. In some implementations, this approach can be combined with learning shared projection matrices that transfer layer-wise knowledge from the teacher language model to the student language model. Example experimental results have also demonstrated higher compression efficiency and accuracy when compared with other state-of-the-art compression techniques, including the ability to compress the BERTmodel by more than 60×, with only a minor drop in downstream task metrics, resulting in a language model with a footprint of under 7 MB.

Extreme language model compression with optimal sub-words and shared projections

A deep state space generative model is augmented with intervention prediction. The state space model provides a principled way to capture the interactions among observations, interventions, critical event occurrences, true states, and associated uncertainty. The state space model can include a discrete-time hazard rate model that provides flexible fitting of general survival time distributions. The state space model can output a joint prediction of event risk, observation and intervention trajectories based on patterns in temporal progressions, and correlations between past measurements and interventions.

Machine-learned state space model for joint forecasting

The present disclosure is directed to systems and methods for performing reading comprehension with machine learning. More specifically, the present disclosure is directed to a Neural Symbolic Reader (example implementations of which may be referred to as NeRd), which includes a reader to encode the passage and question, and a programmer to generate a program for multi-step reasoning. By using operators like span selection, the program can be executed over a natural language text passage to generate an answer to a natural language text question. NeRd is domain-agnostic such that the same neural architecture works for different domains. Further, NeRd is compositional such that complex programs can be generated by compositionally applying the symbolic operators.

Neural symbolic reader

Provided is a knowledge distillation technique for training a student language model that, relative to a larger teacher language model, has a significantly smaller vocabulary, lower embedding dimensions, and/or hidden state dimensions. Specifically, aspects of the present disclosure are directed to a dual-training mechanism that trains the teacher and student language models simultaneously to obtain optimal word embeddings for the student vocabulary. In some implementations, this approach can be combined with learning shared projection matrices that transfer layer-wise knowledge from the teacher language model to the student language model. Example experimental results have also demonstrated higher compression efficiency and accuracy when compared with other state-of-the-art compression techniques, including the ability to compress the BERTmodel by more than 60&#xd7;, with only a minor drop in downstream task metrics, resulting in a language model with a footprint of under 7 MB.

Described are systems, methods, and computer-readable media for program generation in a domain-specific language based on input-output examples. In accordance with various embodiments, a neural-network-based program generation model conditioned on an encoded set of input-output examples is used to generate a program tree by iteratively expanding a partial program tree, beginning with a root node and ending when all leaf nodes are terminal.

Neural network for program synthesis

Architecture that performs image page index selection. A learning-based framework learns a statistical model based on the hyperlink (URL-uniform resource locator) previous click information obtained from the image search users. The learned model can combine the features of a newly discovered URL to predict the possibility of the newly-discovered URL being clicked in the future image search. In addition to existing web index selection features, image clicks are added as features, and the image clicks are aggregated over different URL segments, as well as the site modeling pattern trees to reduce the sparse problem of the image click information.

Learning-based image webpage index selection

A system and method infer true labels for multiple items. The inferred labels are generated from judgments. Multiple judges select the judgments from a specified choice of labels for each item. The method includes determining a characterization of judge expertise and item difficulties based on the judgments. The method also includes determining, using maximum entropy, a probability distribution over the specified choice of labels for each judge and item, based on the judgments. The method further includes selecting improved labels for the items from the specified choice such that the entropy over the probability distribution is reduced. The improved labels represent an improvement from the judgments toward the true labels. Additionally, the method includes performing iterative procedure to determine the true labels, the characterizations of judge expertise and the labeling difficulties.

Learning with noisy labels from multiple judges

A spam detection system is disclosed. The system includes a classifier training component that receives a first set of training pages labeled as normal pages and a second set of training pages labeled as spam pages. The training component trains a web page classifier based on both the first set of training pages and the second set of training pages. A spam detector then receives unlabeled web pages uses the web page classifier to classify the unlabeled web pages as spam pages or normal pages.

Link spam detection using smooth classification function

A method is described herein that includes acts of receiving a selection of a first phrase in a first language and executing a random walk over a computer-implemented multipartite graph, wherein the multipartite-graph includes a first set of nodes that are representative of phrases in the first language, a second set of nodes that are representative of phrases in a second language, and edges between nodes that are representative of relationships between the respective phrases. The random walk includes traversals over edges of the graph between nodes. The method also includes the act of indicating that a second phrase in the first language is a paraphrase of the first phrase based at least in part upon the random walk.

Locating paraphrases through utilization of a multipartite graph

A system described herein includes analyzer component that analyzes queries submitted by users and corresponding URLs selected by the users, wherein the queries include a first query and a second query, and wherein the analyzer component determines that the first query and the second query are substantially similar queries. The system additionally includes a correlator component that, responsive to the analyzer component determining that the first query and the second query are substantially similar, generates correlation data that indicates that the first and second queries are substantially similar.

Substantially similar queries

A collection of web pages is considered as a directed graph in which the pages themselves are nodes and the hyperlinks between the pages are directed edges in the graph. A trusted entity identifies training examples for spam pages and normal pages. A random walk is conducted through the directed graph that includes the collection of web pages and the stationary probabilities, and transitional probabilities, among the nodes in the directed graph are obtained. A classifier training component estimates a classification function that changes slowly on densely connected subgraphs within the directed graph. The classification function assigns a value to each of the nodes in the directed graph and identifies them as spam or normal pages based upon whether the value meets a given function threshold value.

A collection of web pages is modeled as a directed graph, in which the nodes of the graph are the web pages and directed edges are hyperlinks. Web pages can also be represented by content, or by other features, to obtain a similarity graph over the web pages, where nodes again denote the web pages and the links or edges between each pair of nodes is weighted by a corresponding similarity between those two nodes. A random walk is defined for each graph, and a mixture of the random walks is obtained for the set of graphs. The collection of web pages is then analyzed based on the mixture to obtain a web page analysis result. The web page analysis result can be, for example, clustering of the web pages to discover web communities, classifying or categorizing the web pages, or spam detection indicating whether a given web page is spam or content.

Web page analysis using multiple graphs

Growing community of inventors

Redmond, WA, United States of America

Dengyong Zhou