Oracle International Corporation

Adam Pocock

Michael Louis Wick

Vladimir V Zelevinsky

Pallika Haridas Kanani

Katherine Silverstein

Uri Sheffer

Brook Stevens

Mashhood Ishaque

Tristan R Spaulding

Mark Edward Johnson

Thanh Long Duong

Vishal Vishnoi

Srinivasa Phani Kumar Gadde

Cong Duy Vu Hoang

Poorya Zaremoodi

Guy L Steele

Thanh Tien Vu

Jean-Baptiste Frederic George Tristan

Tuyen Quang Pham

Jean-Baptiste Tristan

Duy Vu

Budhaditya Saha

Dai Hoang Tran

Nagaraj N Bhat

A method for training a language model using negative data may include accessing a first training corpus comprising positive training data and accessing a second training corpus comprising negative training data. The method may further include training a first language model using at least the first training corpus, the second training corpus, and a maximum likelihood function. The maximum likelihood function may maximize the likelihood of the first language model predicting the positive training data while minimizing the likelihood of the first language model predicting the negative training data.

Removing undesirable signals from language models using negative data

Techniques are disclosed herein for focused training of language models and end-to-end hypertuning of the framework. In one aspect, a method is provided that includes obtaining a machine learning model pre-trained for language modeling, and post-training the machine learning model for various tasks to generate a focused machine learning model. The post-training includes: (i) training the machine learning model on an unlabeled set of training data pertaining to a task that the machine learning model was pre-trained for as part of the language modeling, and the unlabeled set of training data is obtained with respect to a target domain, a target task, or a target language, and (ii) training the machine learning model on a labeled set of training data that pertains to another task that is an auxiliary task related to a downstream task to be performed using the machine learning model or output from the machine learning model.

Framework for focused training of language models and techniques for end-to-end hypertuning of the framework

Systems and methods are disclosed to build and execute a decision system based on multiple machine learned decision models. In embodiments, the decision system performs a hashing technique to reduce relevant features of the input data into a feature vector for each decision model. The feature vector reduces the dimensionality of the feature universe of the input data, and its use allows the decision models to be trained and executed using less computing resources. In embodiments, the decision system implements an ensembled decision model that makes decisions based on a combination function that combines the decision results of the individual models in the ensemble. The decision models employ different hashing techniques to hash the input features differently, so that errors caused by the feature hashing of individual models are reduced in the aggregate.

Ensembled decision systems using feature hashing models

A system is provided that extracts attribute values. The system receives data including unstructured text from a data store. The system further tokenizes the unstructured text into tokens, where a token is a character of the unstructured text. The system further annotates the tokens with attribute labels, where an attribute label for a token is determined, in least in part, based on a word that the token originates from within the unstructured text. The system further groups the tokens into text segments based on the attribute labels, where a set of tokens that are annotated with an identical attribute label are grouped into a text segment, and where the text segments define attribute values. The system further stores the attribute labels and the attribute values within the data store.

Character-based attribute value extraction system

A system for exploring data receives the data from a database and indexes the data in a server. The system displays one or more selectable datasets from the indexed data, where the selectable datasets include a plurality of attributes. The system receives a selection of one of the plurality of attributes. The system then sorts the one or more attributes by level of interestingness relative to the selected attribute, and displays the sorted attributes.

Guided data exploration

A system for exploring data receives the data from a database and indexes the data in a server. The system displays one or more selectable datasets from the indexed data, where the selected dataset includes one or more attributes. The system then sorts the one or more attributes by level of interestingness and displays the sorted attributes.

A natural language processing ('NLP') manager is provided that manages NLP model training. An unlabeled corpus of multilingual documents is provided that span a plurality of target languages. A multilingual embedding is trained on the corpus of multilingual documents as input training data, the multilingual embedding being generalized across the target languages by modifying the input training data and/or transforming multilingual dictionaries into constraints in an underlying optimization problem. An NLP model is trained on training data for a first language of the target languages, using word embeddings of the trained multilingual embedding as features. The trained NLP model is applied for data from a second of the target languages, the first and second languages being different.

Multilingual embeddings for natural language processing

Growing community of inventors

Burlington, MA, United States of America

Adam Pocock