Salesforce, Inc.

Shiva Kumar Pentyala

Ankit Chadha

Shashank Harinath

Prafulla Kumar Choubey

Mridul Gupta

Richard Socher

Zachary Alexander

Sitaram Asur

Jean-Marc Soumet

Johnson Liu

Shilpa Bhagavath

Abhishek Sharma

Zhichao Wang

Bin Bi

Gurkirat Singh

Suhail Sanjiv Barot

Jingyuan Liu

Indira Iyer

Embodiments described herein provide a training framework for generative NLP models. Specifically, the training input, e.g., in the form of a sequence of tokens representing a user-agent dialogue, may be randomly masked for a few spans, which can be one or more tokens, one or more words, one or more sentences, or one or more paragraphs. These masked spans are replaced with their embeddings generated from pre-trained large language models are then used for training the NLP model.

Systems and methods for training a neural network model using knowledge from pre-trained large language models

Embodiments described herein provide a training framework for generative NLP models that operate on previously learnt knowledge from pretrained large language models. Specifically, to train an NLP model to generate a response to a user utterance (e.g., 'resolve login issue'), document embeddings of support IT documents encoded by a pretrained LLM are fed to an NLP decoder together with a training dialogue (e.g., a dialogue between the chat agent on how to “resolve login issue”). The NLP decoder can thus be trained by a causal language modeling loss computed based on the predicted next token and the ground-truth token from the training dialogue.

In some systems, a set of sentences of a relatively large document may be vectorized into a set of vectors via an embedding model for summarization. Further, a subset of vectors of the set of vectors may be selected via a farthest point sampling (FPS) procedure based on a vector-space distance between respective vectors of the subset of vectors. Moreover, the subset of vectors that are associated with a subset of sentences may be ordered based on the order of the subset of sentences within the set of sentences of the document. Further, to generate a summary of the document, a query may be transmitted to a large language model (LLM) that includes a summarization prompt and the subset of sentences that correspond with the selected subset of vectors. A summary of the document may then be received from the LLM based on transmitting the query.

Extractive-abstractive large language model summarization with farthest point sampling

Embodiments described herein provide an automated testing pipeline for providing a testing dataset for testing a trained neural network model trained using a first training dataset. A first testing dataset for the trained neural network including a first plurality of user queries is received. A dependency parser is used to filter the first plurality of user queries based on one or more action verbs. A pretrained language model is used to rank the remaining user queries based on respective relationships with queries in the first training dataset. Further, user queries that are classified as keyword matches with the queries in the first training dataset using a bag of words classifier are removed. A second testing dataset is generated using the ranked remaining user queries. Testing outputs are generated, by the trained neural network model, using the second testing dataset.

Systems and methods for providing an automated testing pipeline for neural network models

A system performs named entity recognition for performing natural language processing, for example, for conversation engines. The system uses context information in named entity recognition. The system includes the context of a sentence during model training and execution. The system generates high quality contextual data for training NER models. The system utilizes labeled and unlabeled contextual data for training NER models. The system provides NER models for execution in production environments. The system uses heuristics to determine whether to use a context-based NER model or a simple NER model that does not use context information. This allows the system to use simple NER models when the likelihood of improving the accuracy of prediction based on context is low.

Generation of training data for machine learning based models for named entity recognition for natural language processing

Methods and systems for hierarchical natural language understanding are described. A representation of an utterance is inputted to a first machine learning model to obtain information on the first utterance. According to the information on the utterance a determination that the representation of the utterance is to be inputted to a second machine learning model that performs a dedicated natural language task is performed. In response to determining that the representation of the utterance is to be inputted to a second machine learning model, the utterance is inputted to the second machine learning model to obtain an output of the dedicated natural language task.

Hierarchical natural language understanding systems

A natural language processing system that trains task models for particular natural language tasks programmatically generates additional utterances for inclusion in the training set, based on the existing utterances in the training set and the existing state of a task model as generated from the original (non-augmented) training set. More specifically, the training augmentation moduleidentifies specific textual units of utterances and generates variants of the utterances based on those identified units. The identification is based on determined importances of the textual units to the output of the task model, as well as on task rules that correspond to the natural language task for which the task model is being generated. The generation of the additional utterances improves the quality of the task model without the expense of manual labeling of utterances for training set inclusion.

Growing community of inventors

Mountain View, CA, United States of America

Shiva Kumar Pentyala