Oracle International Corporation

Nagaraj N Bhat

Mark Edward Johnson

Thanh Long Duong

Vishal Vishnoi

Srinivasa Phani Kumar Gadde

Cong Duy Vu Hoang

Poorya Zaremoodi

Tuyen Quang Pham

Thanh Tien Vu

Adam Pocock

Duy Vu

Budhaditya Saha

Katherine Silverstein

Dai Hoang Tran

Shahid Reza

Techniques are disclosed herein for focused training of language models and end-to-end hypertuning of the framework. In one aspect, a method is provided that includes obtaining a machine learning model pre-trained for language modeling, and post-training the machine learning model for various tasks to generate a focused machine learning model. The post-training includes: (i) training the machine learning model on an unlabeled set of training data pertaining to a task that the machine learning model was pre-trained for as part of the language modeling, and the unlabeled set of training data is obtained with respect to a target domain, a target task, or a target language, and (ii) training the machine learning model on a labeled set of training data that pertains to another task that is an auxiliary task related to a downstream task to be performed using the machine learning model or output from the machine learning model.

Framework for focused training of language models and techniques for end-to-end hypertuning of the framework

Techniques for dynamically developing a contextual set of prompts based on relevant aspects extracted from s set of training data. One technique includes obtaining training data comprising text examples and associated labels, extracting aspects from the training data, generating prompting templates based on the training data and the extracted aspects, concatenating each of the text examples with the respective generated prompting template to create prompting functions, training a machine learning language model on the prompting functions to predict a solution for a task, where the training is formulated as a masked language modeling problem with blanks of the prompting templates being set as text labels and expected output for the task being set as specified solution labels, and the training learns or updates model parameters of the machine learning language model for performing the task. The machine learning language model is provided with the learned or updated model parameters.

Aspect prompting framework for language modeling

Growing community of inventors

Bengaluru, India

Nagaraj N Bhat