Oracle International Corporation

Katherine Silverstein

Michael Louis Wick

Adam Pocock

Mark Edward Johnson

Thanh Long Duong

Vishal Vishnoi

Srinivasa Phani Kumar Gadde

Cong Duy Vu Hoang

Poorya Zaremoodi

Tuyen Quang Pham

Jean-Baptiste Frederic George Tristan

Thanh Tien Vu

Duy Vu

Pallika Haridas Kanani

Budhaditya Saha

Nagaraj N Bhat

Dai Hoang Tran

A method for training a language model using negative data may include accessing a first training corpus comprising positive training data and accessing a second training corpus comprising negative training data. The method may further include training a first language model using at least the first training corpus, the second training corpus, and a maximum likelihood function. The maximum likelihood function may maximize the likelihood of the first language model predicting the positive training data while minimizing the likelihood of the first language model predicting the negative training data.

Removing undesirable signals from language models using negative data

Techniques are disclosed herein for focused training of language models and end-to-end hypertuning of the framework. In one aspect, a method is provided that includes obtaining a machine learning model pre-trained for language modeling, and post-training the machine learning model for various tasks to generate a focused machine learning model. The post-training includes: (i) training the machine learning model on an unlabeled set of training data pertaining to a task that the machine learning model was pre-trained for as part of the language modeling, and the unlabeled set of training data is obtained with respect to a target domain, a target task, or a target language, and (ii) training the machine learning model on a labeled set of training data that pertains to another task that is an auxiliary task related to a downstream task to be performed using the machine learning model or output from the machine learning model.

Framework for focused training of language models and techniques for end-to-end hypertuning of the framework

A system that performs natural language processing receives a text corpus that includes a plurality of documents and receives a knowledge base. The system generates a set of document n-grams from the text corpus and considers all n-grams as candidate mentions. The system, for each candidate mention, queries the knowledge base and in response retrieves results. From the results retrieved by the queries, the system generates a search space and generates a joint model from the search space.

Named entity recognition and entity linking joint training

Growing community of inventors

Amherst, MA, United States of America

Katherine Silverstein