Microsoft Technology Licensing, LLC

Hoifung Poon

Christopher Brian Quirk

Kristina Nikolova Toutanova

Jianfeng Gao

Yu Wang

Pengcheng He

Wen-tau Yih

Hunter Lang

Hao Cheng

Cliff Richard Wong

Xiaodong Liu

Weizhu Chen

Colin A Cherry

Aditya Vithal Nori

Sheng Zhang

Daniel Coelho De Castro

Anton Schwaighofer

Ozan Oktay

Ankur P Parikh

Javier Alvarez-valle

Hai Wang

Nanyun Peng

Melanie Bernhardt

Naoto Usuyama

Sarthak Jain

Tristan Josef Naumann

Robin Jia

Shruthi Jaisimha Bannur

This document relates to training of machine learning models. One example method involves providing a machine learning model having one or more mapping layers. The one or more mapping layers can include at least a first mapping layer configured to map components of pretraining examples into first representations in a space. The example method also includes performing a pretraining stage on the one or more mapping layers using the pretraining examples. The pretraining stage can include adding noise to the first representations of the components of the pretraining examples to obtain noise-adjusted first representations. The pretraining stage can also include performing a self-supervised learning process to pretrain the one or more mapping layers using at least the first representations of the training data items and the noise-adjusted first representations of the training data items.

Adversarial pretraining of machine learning models

Systems and methods are provided for generating and training a relation extraction model configured to extract document-level relations. Systems obtain a knowledge database that comprises a plurality of entity tuples and a plurality of relation types, use the knowledge database to generate annotated relation instances based on relation instances that are identified in a set of unlabeled text, generate a training dataset comprising the annotated relation instances and the set of unlabeled text, and generate the machine learning model via modular self-supervision. Systems and methods are also provided for using a relation extraction model to extract document-level relations in specific use scenarios, such as for extracting drug response relations from full-text medical research articles.

Modular self-supervision for document-level relation extraction

The present disclosure relates to devices and methods for determining new virtual evidence to use with a deep probabilistic logic module. The devices and methods may receive output from a deep probabilistic logic module in response to running an initial set of virtual evidence through the deep probabilistic logic module. The devices and methods may use the output to automatically propose at least one factor as new virtual evidence for use with the deep probabilistic logic module. The devices and methods may add the new virtual evidence to the deep probabilistic logic module.

Self-supervised self supervision by combining probabilistic logic with deep learning

A computer implemented method comprising: receiving a report on a condition of a human or animal subject, composed by a user based on a scan of the subject; inputting the current report and the scan into a trained machine learning model; and based on the report and the scan, the machine learning model generating one or more suggestions for updating the text of the report. The method further comprises causing a user interface to display to the user one or more suggestions for updating the text of the report, with each respective suggestion visually linked in the user interface to a corresponding subregion within at least one image of the scan based upon which the respective suggestion was generated.

Generating reports from scanned images

Systems, methods, and computer-readable media for providing entity relation extraction across sentences in a document using distant supervision are disclosed. A computing device can receive an input, such as a document comprising a plurality of sentences. The computing device can identify syntactic and/or semantic links between words in a sentence and/or between words in different sentences, and extract relationships between entities throughout the document. A knowledge base (e.g., a table, chart, database etc.) of entity relations based on the extracted relationships can be populated. An output of the populated knowledge base can be used by a classifier to identify additional relationships between entities in various documents. Machine learning can be applied to train the classifier to predict relations between entities. The classifier can be trained using known entity relations, syntactic links and/or semantic links.

Relation extraction across sentence boundaries

A technique is described herein for processing documents in a time-efficient and accurate manner. In a training phase, the technique generates a set of initial training examples by associating entity mentions in a text corpus with corresponding entity identifiers. Each entity identifier uniquely identifies an entity in a particular ontology. The technique then removes noisy training examples from the set of initial training examples, to provide a set of filtered training examples. The technique then applies a machine-learning process to generate a linking component based, in part, on the set of filtered training examples. In an application phase, the technique uses the linking component to link input entity mentions with corresponding entity identifiers. Various application systems can leverage the capabilities of the linking component, including a search system, a document-creation system, etc.

Distant supervision for entity linking with filtering of noise

Systems, methods, and computer-readable media provide entity relation extraction across sentences in a document using distant supervision. A computing device can receive an input, such as a document comprising a plurality of sentences. The computing device can identify syntactic and/or semantic links between words in a sentence and/or between words in different sentences, and extract relationships between entities throughout the document. A knowledge base (e.g., a table, chart, database etc.) of entity relations based on the extracted relationships can be populated. An output of the populated knowledge base can be used by a classifier to identify additional relationships between entities in various documents. Machine learning can be applied to train the classifier to predict relations between entities. The classifier can be trained using known entity relations, syntactic links and/or semantic links.

A computing system is provided. The computing system includes a processor configured to execute one or more programs and associated memory. The processor is configured to execute neural network system that includes a first neural network and a second neural network. The processor is configured to receive input text, and for each of a plurality of text spans within the input text: identify a vector of semantic entities and a vector of entity mentions; define an n-ary relation between entity mentions including subrelations; and determine mention-level representation vectors in the text spans that satisfy the n-ary relation or subrelations. The processor is configured to: aggregate the mention-level representation vectors over all of the text spans to produce entity-level representation vectors; input to the second neural network the entity-level representation vectors; and output a prediction of a presence of the n-ary relation for the semantic entities in the input text.

N-ary relation prediction over text spans

Long short term memory units that accept a non-predefined number of inputs are used to provide natural language relation extraction over a user-specified range on content. Content written for human consumption is parsed with distant supervision in segments (e.g., sentences, paragraphs, chapters) to determine relationships between various words within and between those segments.

Graph long short term memory for syntactic relationship discovery

The system that performs semantic parsing may automatically extract complex information from databases. Complex information may comprise nested event structures. In one example process, a processor may receive unannotated text and may access a natural-language database that includes nested events. The processor, in performing semantic parsing, may automatically generate syntactic trees that include annotations that represent the semantic information. In particular, the natural-language sentences and the database include nested event structures.

Semantic parsing for complex knowledge extraction

Described is a technology for performing unsupervised learning using global features extracted from unlabeled examples. The unsupervised learning process may be used to train a log-linear model, such as for use in morphological segmentation of words. For example, segmentations of the examples are sampled based upon the global features to produce a segmented corpus and log-linear model, which are then iteratively reprocessed to produce a final segmented corpus and a log-linear model.

Growing community of inventors

Bellevue, WA, United States of America

Hoifung Poon