Oracle International Corporation

Gioacchino Tangari

Thanh Long Duong

Cong Duy Vu Hoang

Aashna Devang Kanuga

Nitika Mathur

Mark Edward Johnson

Vishal Vishnoi

Philip Arthur

Srinivasa Phani Kumar Gadde

Balakota Srinivas Vinnakota

Poorya Zaremoodi

Steve Wai-Chun Siu

Yakupitiyage Don Thanuja Samodhye Dharmasiri

Jae Min John

Shivashankar Subramanian

Syed Najam Abbas Zaidi

Chang Xu

Techniques are disclosed herein for improving model robustness on operators and triggering keywords in natural language to a meaning representation language system. The techniques include augmenting an original set of training data for a target robustness bucket by leveraging a combination of two training data generation techniques: (1) modification of existing training examples and (2) synthetic template-based example generation. The resulting set of augmented data examples from the two training data generation techniques are appended to the original set of training data to generate an augmented training data set and the augmented training data set is used to train a machine learning model to generate logical forms for utterances.

Model robustness on operators and triggering keywords in natural language to a meaning representation language system

Techniques are disclosed herein for calibrating confidence scores of a machine learning model trained to translate natural language to a meaning representation language. The techniques include obtaining one or more raw beam scores generated from one or more beam levels of a decoder of a machine learning model trained to translate natural language to a logical form, where each of the one or more raw beam scores is a conditional probability of a sub-tree determined by a heuristic search algorithm of the decoder at one of the one or more beam levels, classifying, by a calibration model, a logical form output by the machine learning model as correct or incorrect based on the one or more raw beam scores, and providing the logical form with a confidence score that is determined based on the classifying of the logical form.

Calibrating confidence scores of a machine learning model trained as a natural language interface

Techniques for preprocessing data assets to be used in a natural language to logical form model based on scalable search and content-based schema linking. In one particular aspect, a method includes accessing an utterance, classifying named entities within the utterance into predefined classes, searching value lists within the database schema using tokens from the utterance to identify and output value matches including: (i) any value within the value lists that matches a token from the utterance and (ii) any attribute associated with a matching value, generating a data structure by organizing and storing: (i) each of the named entities and an assigned class for each of the named entities, (ii) each of the value matches and the token matching each of the value matches, and (iii) the utterance, in a predefined format for the data structure, and outputting the data structure.

Transforming natural language to structured query language based on scalable search and content-based schema linking

Techniques are disclosed herein for synthesizing synthetic training data to facilitate training a natural language to logical form model. In one aspect, training data can be synthesized from original under a framework based on templates and a synchronous context-free grammar. In one aspect, training data can be synthesized under a framework based on a probabilistic context-free grammar and a translator. In one aspect, training data can be synthesized under a framework based on tree-to-string translation. In one aspect, the synthetic training data can be combined with original training data in order to train a machine learning model to translate an utterance to a logical form.

Data manufacturing frameworks for synthesizing synthetic training data to facilitate training a natural language to logical form model

Growing community of inventors

Sydney, Australia

Gioacchino Tangari