Microsoft Technology Licensing, LLC

Ruofei Zhang

Keng-hao Chang

Jian Jiao

Changbo Hu

Qun Li

Zi Yin

Jianfeng Gao

Nan Duan

Yeyun Gong

Muhammad Adnan Alam

Qiang Lou

John Weston Hughes

Tianchuan Du

Paul Liu

Li Deng

Ming Zhou

Bin Gao

Xiaodong He

Xiaodong Liu

Min Li

Yelong Shen

Guowei Liu

Jia Le He

Srinivasa Reddy Neerudu

Jianchang Mao

Gregory T Buehrer

Shuangfei Zhai

Yunhong Zhou

Sujith Vishwajith

Luis Gerardo Mojica De La Vega

Mu-Chu Lee

Xiaozong Wang

Javad Azimi

Pengqi Liu

Wenhao Lu

A tagging system appends supplemental information to an original sequence of items, to produce a supplemented sequence of items. The tagging system includes a transformer-based encoder neural network that maps the supplemented sequence into hidden state information. The tagging system includes a post-processing neural network that transform the hidden state information into a tagged output sequence of items. That is, each item in the tagged output sequence includes a tag that identifies its entity class or some other characteristic. The tagging system can increase the accuracy of the tags it produces by virtue of the inclusion of the supplemental information added to each original sequence. A training system trains the tagging system to perform plural tasks, which further increases the accuracy of the tags it produces. The training system may commence training of the tagging system using a pre-trained model for the encoder neural network.

Multi-task sequence tagging with injection of supplemental information

A transformer-based neural network includes at least one mask attention network (MAN). The MAN computes an original attention data structure that expresses influence between pairs of data items in a sequence of data items. The MAN then modifies the original data structure by mask values in a mask data structure, to produce a modified attention data structure. Compared to the original attention data structure, the modified attention data structure better accounts for the influence of neighboring data items in the sequence of data items, given a particular data item under consideration. The mask data structure used by the MAN can have static and/or machine-trained mask values. In one implementation, the transformer-based neural network includes at least one MAN in combination with at least one other attention network that does not use a mask data structure, and at least one feed-forward neural network.

Transformer-based neural network including a mask attention network

A training system produces a resource-efficient machine-trained model via a training architecture that employs plural processing paths. Some of the processing paths incorporate the use of auxiliary information that imparts external knowledge about source items being processed. The training architecture also employs contrastive learning that operates at different respective levels within the training architecture. For instance, the training architecture uses encoder-level contrastive learning to compare output information generated by different encoders within the training architecture. The training architecture uses decoder-level contrastive learning to compare output information produced by different decoders within the training architecture. An inference-stage system performs an application task using the model produced by the training system.

Resource-efficient sequence generation with dual-level contrastive learning

Neural network-based categorization can be improved by incorporating graph neural networks that operate on a graph representing the taxonomy of the categories into which a given input is to be categorized by the neural network based-categorization. The output of a graph neural network, operating on a graph representing the taxonomy of categories, can be combined with the output of a neural network operating upon the input to be categorized, such as through an interaction of multidimensional output data, such as a dot product of output vectors. In such a manner, information conveying the explicit relationships between categories, as defined by the taxonomy, can be incorporated into the categorization. To recapture information, incorporate new information, or reemphasize information a second neural network can also operate upon the input to be categorized, with the output of such a second neural network being merged with the output of the interaction.

Neural network categorization accuracy with categorical graph neural networks

Described herein are technologies related to constructing supplemental content items that summarize electronic landing pages. A sequence to sequence model that is configured to construct supplemental content items is trained based upon a corpus of electronic landing pages and supplemental content items that have been constructed by domain experts, wherein each landing page has a respective supplemental content item assigned thereto. The sequence to sequence model is additionally trained using self critical sequence training, where estimated click through rates of supplemental content items generated by the sequence to sequence model are employed to train the sequence to sequence model.

Generating electronic summary documents for landing pages

Knowledge graphs can greatly improve the quality of content recommendation systems. There is a broad variety of knowledge graphs in the domain including clicked user-ad graphs, clicked query-ad graphs, keyword-display URL graphs etc. A hierarchical Transformer model learns entity embeddings in knowledge graphs. The model consists of two different Transformer blocks where the bottom block generates relation-dependent embeddings for the source entity and its neighbors, and the top block aggregates the outputs from the bottom block to produce the target entity embedding. To balance the information from contextual entities and the source entity itself, a masked entity model (MEM) task is combined with a link prediction task in model training.

Learning graph representations using hierarchical transformers for content recommendation

The present application describes a system and method for converting a natural language query to a standard query using a sequence-to-sequence neural network. As described herein, when a natural language query is receive, the natural language query is converted to a standard query using a sequence-to-sequence model. In some cases, the sequence-to-sequence model is associated with an attention layer. A search using the standard query is performed and various documents may be returned. The documents that result from the search are scored based, at least in part, on a determined conditional entropy of the document. The conditional entropy is determined using the natural language query and the document.

Query rewriting and interactive inquiry framework

A technique is described herein for processing a given query item in a latency-efficient and resource-efficient manner. The technique uses a first transformer-based encoder to transform the given query item into an encoded query item. In one case, the given query item is an expression that includes one or more query-expression linguistic tokens. The technique includes a second transformer-based encoder for transforming a given target item into an encoded target item. The given target item may likewise correspond to an expression that includes one or more target-expression linguistic tokens. A similarity-assessing mechanism then assesses the semantic similarity between the given query item and the given target item based on the encoded query item and the encoded target item. Each transformer-based encoder uses one or more self-attention mechanisms. The second transformer-based encoder can optionally perform its work in an offline manner, prior to receipt of the given query item.

Assessing semantic similarity using a dual-encoder neural network

A computer-implemented technique uses one or more neural networks to identify at least one item name associated with an input image using a multi-modal fusion approach. The technique is said to be multi-modal because it collects and processes different kinds of evidence regarding each detected item name. The technique is said to adopt a fusion approach because it fuses the multi-modal evidence into an output conclusion that identifies at least one item name associated with the input image. In one example, a first mode collects evidence by identifying and analyzing regions in the input image that are likely to include item name-related information. A second mode collects and analyzes any text that appears as part of input image itself. A third mode collects and analyzes text that is not included in the input image itself, but is nonetheless associated with the input image.

Processing image-bearing electronic documents using a multimodal fusion framework

A computer-implemented technique uses a generative adversarial network (GAN) to jointly train a generator neural network ('generator') and a discriminator neural network ('discriminator'). Unlike traditional GAN designs, the discriminator performs the dual role of: (a) determining one or more attribute values associated with an object depicted in input image fed to the discriminator; and (b) determining whether the input image fed to the discriminator is real or synthesized by the generator. Also unlike traditional GAN designs, an image classifier can make use of a model produced by the GAN's discriminator. The generator receives generator input information that includes a conditional input image and one or more conditional values that express desired characteristics of the generator output image. The discriminator receives the conditional input image in conjunction with a discriminator input image, which corresponds to either the generator output image or a real image.

Multi-task GAN, and image translator and image classifier trained thereby

A taxonomy of categories, attributes, and values can be conflated with new data triplets by identifying one or more conflation candidates among the attribute-value pairs within a category of the taxonomy that matches the category of the data triplet, and determining a suitable merge action for conflating the data triplet with each conflation candidate. The task of determining merge actions may be cast as a classification problem, and may be solved by an ensemble classifier.

Taxonomy enrichment using ensemble classifiers

Technologies are described herein that relate to identifying supplemental content items that are related to objects captured in images of webpages. A computing system receives an indication that a client computing device has a webpage displayed thereon that includes an image. The image is provided to a first DNN that is configured to identify a portion of the image that includes an object of a type from amongst a plurality of predefined types. Once the portion of the image is identified, the portion of the image is provided to a plurality of DNNs, with each of the DNNs configured to output a word or phrase that represents a value of a respective attribute of the object. A sequence of words or phrases output by the plurality of DNNs is provided to a search computing system, which identifies a supplemental content item based upon the sequence of words or phrases.

Pipeline for identifying supplemental content items that are related to objects in images

Growing community of inventors

Mountain View, CA, United States of America

Ruofei Zhang