Salesforce, Inc.

Prafulla Kumar Choubey

Nazneen Rajani

Chien-Sheng Wu

Jesse Vig

Alexander R Fabbri

Caiming Xiong

Wenhao Liu

Shiva Kumar Pentyala

Benjamin Newman

Embodiments described herein provide a training framework for generative NLP models that operate on previously learnt knowledge from pretrained large language models. Specifically, to train an NLP model to generate a response to a user utterance (e.g., 'resolve login issue'), document embeddings of support IT documents encoded by a pretrained LLM are fed to an NLP decoder together with a training dialogue (e.g., a dialogue between the chat agent on how to “resolve login issue”). The NLP decoder can thus be trained by a causal language modeling loss computed based on the predicted next token and the ground-truth token from the training dialogue.

Systems and methods for training a neural network model using knowledge from pre-trained large language models

Embodiments described herein provide a document summarization framework that employs an ensemble of summarization models, each of which is a modified version of a base summarization model to control hallucination. For example, a base summarization model may first be trained on a full training data set. The trained base summarization model is then fine-tuned using a first filtered subset of the training data which contains noisy data, resulting in an 'anti-expert' model. The parameters of the anti-expert model are subtracted from the parameters of the trained base model to produce a final summarization model which yields robust factual performance.

Systems and methods for parameter ensembling for reducing hallucination in abstractive summarization

Embodiments described herein provide a method for text summarization. The method includes receiving a training dataset having at least an uncompressed text, a compressed text, and one or more information entities accompanying the compressed text. The method also includes generating, using a perturber model, a perturbed text with the one or more information entities being inserted into the compressed text. The method further includes training the perturber model based on a first training objective, and generating, using the trained perturber model, a perturbed summary in response to an input of a reference summary. The method further includes generating, via an editor model, a predicted summary by removing information from the perturbed summary conditioned on a source document of the reference summary, and training the editor model based on a second training objective.

Systems and methods for text summarization

Embodiments described herein provide a system and method for extracting factual information. The system transforms a query into a natural language prompt in a format of a query subject and a queried relation. The system encodes, via an embedding layer of a pre-trained language model, the natural language prompt into a first embedding. The system encodes, via the adapter model, the first embedding into a second embedding based on a probability that the second embedding returns the factual information when the second embedding is fed the first attention layer of the pre-trained language model. The system decodes, by the first attention layer of the pre-trained language mode, the second embedding into a response to the query. The system extracts the factual information from the decoded response to the query.

Growing community of inventors

San Jose, CA, United States of America

Prafulla Kumar Choubey