Salesforce, Inc.

Salesforce Com, Inc.

Shashank Harinath

Caiming Xiong

Zachary Alexander

Sitaram Asur

Govardana Sachithanandam Ramachandran

Jean-Marc Soumet

Shiva Kumar Pentyala

Wenhao Liu

Alexis Roos

Michael Machado

Bryan McCann

Qingqing Liu

Na Cheng

Chien-Sheng Wu

Roojuta Lalani

Karl Ryszard Skucha

Thomas Archie Cook, Jr

Zineb Laraki

Manju Vijayakumar

Prafulla Kumar Choubey

Ka Chun Au

Mike Rosenbaum

John Ball

Shilpa Bhagavath

Wojciech Kryscinski

Ankit Chadha

Alexander R Fabbri

Johnson Liu

Jacob Nathaniel Huffman

Victor Yee

Abhishek Sharma

Shubham Mehrotra

Sameer Abhinkar

Nicholas Feinig

Yufan Xue

Linwei Zhu

Embodiments described herein provide a training framework for generative NLP models. Specifically, the training input, e.g., in the form of a sequence of tokens representing a user-agent dialogue, may be randomly masked for a few spans, which can be one or more tokens, one or more words, one or more sentences, or one or more paragraphs. These masked spans are replaced with their embeddings generated from pre-trained large language models are then used for training the NLP model.

Systems and methods for training a neural network model using knowledge from pre-trained large language models

Embodiments described herein provide a training framework for generative NLP models that operate on previously learnt knowledge from pretrained large language models. Specifically, to train an NLP model to generate a response to a user utterance (e.g., 'resolve login issue'), document embeddings of support IT documents encoded by a pretrained LLM are fed to an NLP decoder together with a training dialogue (e.g., a dialogue between the chat agent on how to “resolve login issue”). The NLP decoder can thus be trained by a causal language modeling loss computed based on the predicted next token and the ground-truth token from the training dialogue.

Embodiments described herein provide a structured conversation summarization framework. A user interface may be provided which allows an agent to perform a conversation with a customer, for example regarding resolving a customer support issue. Utterances by both the agent and customer may be stored, and at the end of the conversation, the utterances may be used to generate a structured summary. The structured summary may include components such as a general summary, an issue summary, and a resolution summary. Using neural network models and heuristics, each component of the summary may be automatically generated.

Systems and methods for structured conversation summarization

Embodiments described herein provide an automated testing pipeline for providing a testing dataset for testing a trained neural network model trained using a first training dataset. A first testing dataset for the trained neural network including a first plurality of user queries is received. A dependency parser is used to filter the first plurality of user queries based on one or more action verbs. A pretrained language model is used to rank the remaining user queries based on respective relationships with queries in the first training dataset. Further, user queries that are classified as keyword matches with the queries in the first training dataset using a bag of words classifier are removed. A second testing dataset is generated using the ranked remaining user queries. Testing outputs are generated, by the trained neural network model, using the second testing dataset.

Systems and methods for providing an automated testing pipeline for neural network models

A method that includes receiving an input at an interactive conversation service that uses an intent classification model. The method may further include generating, using an encoder model of the intent classification model, a set of output vectors corresponding to the input, where the encoder model is configured to determine a set of metrics corresponding to intent classifications. The method may further include determining, using an outlier detection model of the intent classification model, whether the input is in-domain or out-of-domain (OOD) based on a first vector of the set of output vectors satisfying a domain threshold relative to one or more of the intent classifications. The method may further include outputting, by the intent classification model, a second vector of the set of output vectors that indicates the set of metrics corresponding to the intent classifications or an indication that the input is OOD.

Multi-lingual intent model with out-of-domain detection

Embodiments described herein provides a training mechanism that transfers the knowledge from a trained BERT model into a much smaller model to approximate the behavior of BERT. Specifically, the BERT model may be treated as a teacher model, and a much smaller student model may be trained using the same inputs to the teacher model and the output from the teacher model. In this way, the student model can be trained within a much shorter time than the BERT teacher model, but with comparable performance with BERT.

Systems and methods for distilled BERT-based training model for text classification

Methods and systems for hierarchical natural language understanding are described. A representation of an utterance is inputted to a first machine learning model to obtain information on the first utterance. According to the information on the utterance a determination that the representation of the utterance is to be inputted to a second machine learning model that performs a dedicated natural language task is performed. In response to determining that the representation of the utterance is to be inputted to a second machine learning model, the utterance is inputted to the second machine learning model to obtain an output of the dedicated natural language task.

Hierarchical natural language understanding systems

Approaches to using unstructured input to update heterogeneous data stores include receiving unstructured text input, receiving a template for interpreting the unstructured text input, identifying, using an entity classifier, entities in the unstructured text input, identifying one or more potential parent entities from the identified entities based on the template, receiving a selection of a parent entity from the one or more potential parent entities, identifying one or more potential child entities from the identified entities based on the template and the selected parent entity, receiving a selection of a child entity from the one or more potential child entities, identifying an action item in the unstructured text input based on the identified entities and the template, determining, using an intent classifier, an intent of the action item, and updating a data store based on the determined intent, the identified entities, and the selected child entity.

Using unstructured input to update heterogeneous data stores

An embodiment proposed herein uses sparsification techniques to train the neural network with a high feature dimension that may yield desirable in-domain detection accuracy but may prune away dimensions in the output that are less important. Specifically, a sparsification vector is generated based on Gaussian distribution (or other probabilistic distribution) and is used to multiply with the higher dimension output to reduce the number of feature dimensions. The pruned output may be then used for the neural network to learn the sparsification vector. In this way, out-of-distribution detection accuracy can be improved.

Systems and methods for out-of-distribution classification

An embodiment provided herein preprocesses the input samples to the classification neural network, e.g., by adding Gaussian noise to word/sentence representations to make the function of the neural network satisfy Lipschitz property such that a small change in the input does not cause much change to the output if the input sample is in-distribution. Method to induce properties in the feature representation of neural network such that for out-of-distribution examples the feature representation magnitude is either close to zero or the feature representation is orthogonal to all class representations. Method to generate examples that are structurally similar to in-domain and semantically out-of domain for use in out-of-domain classification training. Method to prune feature representation dimension to mitigate long tail error of unused dimension in out-of-domain classification. Using these techniques, the accuracy of both in-domain and out-of-distribution identification can be improved.

A method for natural language processing includes receiving, by one or more processors, an unstructured text input. An entity classifier is used to identify entities in the unstructured text input. The identifying the entities includes generating, using a plurality of sub-classifiers of a hierarchical neural network classifier of the entity classifier, a plurality of lower-level entity identifications associated with the unstructured text input. The identifying the entities further includes generating, using a combiner of the hierarchical neural network classifier, a plurality of higher-level entity identifications associated with the unstructured text input based on the plurality of lower-level entity identifications. Identified entities are provided based on the plurality of higher-level entity identifications.

Systems and methods for named entity recognition

Growing community of inventors

San Francisco, CA, United States of America

Shashank Harinath