Microsoft Technology Licensing, LLC

Chenyan Xiong

Paul Nathan Bennett

Saurabh Kumar Tiwary

Xia Song

Junaid Ahmed

Arnold Overwijk

Daniel Fernando Campos

Li Xiong

Corbin Louis Rosset

Nicholas Eric Craswell

Chuan Hu

Ronny Lempel

Corby Louis Rosset

Chen Zhao

Nicholas Craswell

Charumathi Lakshmanan

Jiaming Guo

Jiguang Shen

Ye Li

Hongfei Zhang

A method for text compression comprises recognizing a prefix string of one or more text characters preceding a target string of a plurality of text characters to be compressed. The prefix string is provided to a natural language generation (NLG) model configured to output one or more predicted continuations each having an associated rank. If the one or more predicted continuations include a matching predicted continuation relative to the next one or more text characters of the target string, the next one or more text characters are compressed as an NLG-type compressed representation. If no predicted continuations match the next one or more text characters of the target string, a longest matching entry in a compression dictionary is identified. The next one or more text characters of the target string are compressed as a dictionary-type compressed representation that includes the dictionary index value of the longest matching entry.

Text compression with predicted continuations

A computer-implemented technique is described herein for assisting a user in advancing a task objective. The technique uses a suggestion-generating system (SGS) to provide one or more suggestions to a user in response to at least a last-submitted query provided by the user. The SGS may correspond to a classification-type or generative-type neural network. The SGS uses a machine-trained model that is trained using a multi-task training framework based on plural groups of training examples, which, in turn, are produced using different respective example-generating methods. One such example-generating method constructs a training example from queries in a search session. It operates by identifying the task-related intent the queries, and then identifying at least one sequence of queries in the search session that exhibits a coherent task-related intent. A training example is constructed based on queries in such a sequence.

Using a multi-task-trained neural network to guide interaction with a query-processing system via useful suggestions

This document relates to natural language processing using a framework such as a neural network. One example method involves obtaining a first document and a second document and propagating attention from the first document to the second document. The example method also involves producing contextualized semantic representations of individual words in the second document based at least on the propagating. The contextualized semantic representations can provide a basis for performing one or more natural language processing operations.

Inter-document attention mechanism

Document embedding vectors for each document of a corpus may be generated by combining embedding vectors for document subparts, thereby yielding a final embedding vector for the document. A machine learning model is trained using a query corpus and the document corpus, where the model generates a ranking score for a given (query, document) pair. During training, rankings scores are generated using the model, such that the training dataset is further refined using the generated ranking scores. For example, top documents and a negative document may be determined for a given query and subsequently used as training data. Multiple negative documents may therefore be determined for a given query. A negative document for a given query may be determined from the negative documents using noise-contrastive estimation. Such determined negative documents may be evaluated using a loss function during model training, thereby yielding a more robust model for search processing.

Document body vectorization and noise-contrastive training

To provide automated categorization of structured textual content individual nodes of textual content, from a document object model encapsulation of the structured textual content, have a multidimensional vector associated with them, where the values of the various dimensions of the multidimensional vector are based on the textual content in the corresponding node, the visual features applied or associated with the textual content of the corresponding node, and positional information of the textual content of the corresponding node. The multidimensional vectors are input to a neighbor-imbuing neural network. The enhanced multidimensional vectors output by the neighbor-imbuing neural network are then be provided to a categorization neural network. The resulting output can be in the form of multidimensional vectors whose dimensionality is proportional to categories into which the structured textual content is to be categorized. A weighted merge takes into account multiple nodes that are grouped together.

Automated structured textual content categorization accuracy with neural networks

A system for extracting a key phrase from a document includes a neural key phrase extraction model ('BLING-KPE') having a first layer to extract a word sequence from the document, a second layer to represent each word in the word sequence by ELMo embedding, position embedding, and visual features, and a third layer to concatenate the ELMo embedding, the position embedding, and the visual features to produce hybrid word embeddings. A convolutional transformer models the hybrid word embeddings to n-gram embeddings, and a feedforward layer converts the n-gram embeddings into a probability distribution over a set of n-grams and calculates a key phrase score of each n-gram. The neural key phrase extraction model is trained on annotated data based on a labeled loss function to compute cross entropy loss of the key phrase score of each n-gram as compared with a label from the annotated dataset.

Keyphase extraction beyond language modeling

Keyphrase extraction beyond language modeling

A computer-implemented technique receives an input expression that a user submits with an intent to accomplish some objective. The technique then uses a machine-trained intent encoder component to map the input expression into an input expression intent vector (IEIV). The IEIV corresponds to a distributed representation of the intent associated with the input expression, within a vector intent vector space. The technique then leverages the intent vector to facilitate some downstream application task, such as the retrieval of information. Some application tasks also use a neighbor search component to find expressions that express an intent similar to that of the input expression. A training system trains the intent encoder component based on the nexus between queries and user clicks, as recorded in a search engine's search log.

Intent encoder trained using search logs

Growing community of inventors

Bellevue, WA, United States of America

Chenyan Xiong