Salesforce, Inc.

Salesforce Com, Inc.

Sitaram Asur

Zachary Alexander

Na Cheng

Shiva Kumar Pentyala

Caiming Xiong

Anuprit Kale

Shashank Harinath

Jacob Nathaniel Huffman

Anjan Goswami

Aditya Sakhuja

Aron Kale

Naren M Chittar

Chien-Sheng Wu

Yixin Mao

Prafulla Kumar Choubey

Regunathan Radhakrishnan

Khoa Le

Feifei Jiang

Hui S Fisher

Tiffany Deiandra McKenzie

Alampallam R Ramachandran

Victor Yee

Yuanxin Wang

Jayesh Govindarajan

Weiping Peng

Wojciech Kryscinski

Nicholas Beng Tek Geh

Peter Francis White

Edgar Gerardo Velasco

Alexander R Fabbri

Pingping Xiu

Gary Brandeleer

Kavya Murali

Bin Bi

Zhichao Wang

Fermin Ordaz

Nicholas Feinig

Sameer Abhinkar

Suhas Satish

Ziwei Chen

Embodiments described herein provide a training framework for generative NLP models. Specifically, the training input, e.g., in the form of a sequence of tokens representing a user-agent dialogue, may be randomly masked for a few spans, which can be one or more tokens, one or more words, one or more sentences, or one or more paragraphs. These masked spans are replaced with their embeddings generated from pre-trained large language models are then used for training the NLP model.

Systems and methods for training a neural network model using knowledge from pre-trained large language models

Embodiments described herein provide a training framework for generative NLP models that operate on previously learnt knowledge from pretrained large language models. Specifically, to train an NLP model to generate a response to a user utterance (e.g., 'resolve login issue'), document embeddings of support IT documents encoded by a pretrained LLM are fed to an NLP decoder together with a training dialogue (e.g., a dialogue between the chat agent on how to “resolve login issue”). The NLP decoder can thus be trained by a causal language modeling loss computed based on the predicted next token and the ground-truth token from the training dialogue.

In some systems, a set of sentences of a relatively large document may be vectorized into a set of vectors via an embedding model for summarization. Further, a subset of vectors of the set of vectors may be selected via a farthest point sampling (FPS) procedure based on a vector-space distance between respective vectors of the subset of vectors. Moreover, the subset of vectors that are associated with a subset of sentences may be ordered based on the order of the subset of sentences within the set of sentences of the document. Further, to generate a summary of the document, a query may be transmitted to a large language model (LLM) that includes a summarization prompt and the subset of sentences that correspond with the selected subset of vectors. A summary of the document may then be received from the LLM based on transmitting the query.

Extractive-abstractive large language model summarization with farthest point sampling

Embodiments described herein provide a structured conversation summarization framework. A user interface may be provided which allows an agent to perform a conversation with a customer, for example regarding resolving a customer support issue. Utterances by both the agent and customer may be stored, and at the end of the conversation, the utterances may be used to generate a structured summary. The structured summary may include components such as a general summary, an issue summary, and a resolution summary. Using neural network models and heuristics, each component of the summary may be automatically generated.

Systems and methods for structured conversation summarization

Embodiments described herein provide systems and methods for document recommendation. A system receives a set of training data including a plurality of documents. The system determines whether the set of training data includes annotated contextual information corresponding to the plurality of documents. The system trains supervised and/or unsupervised models based on the availability of data. The models are used to generate vectors representing the documents. During a live text conversation, text from the conversation may be vectorized using the models and the vectors compared to those representing the documents in order to find the most relevant documents. The system may generate an indication of a recommended document.

Systems and methods for real-time document recommendation

Embodiments described herein provide an automated testing pipeline for providing a testing dataset for testing a trained neural network model trained using a first training dataset. A first testing dataset for the trained neural network including a first plurality of user queries is received. A dependency parser is used to filter the first plurality of user queries based on one or more action verbs. A pretrained language model is used to rank the remaining user queries based on respective relationships with queries in the first training dataset. Further, user queries that are classified as keyword matches with the queries in the first training dataset using a bag of words classifier are removed. A second testing dataset is generated using the ranked remaining user queries. Testing outputs are generated, by the trained neural network model, using the second testing dataset.

Systems and methods for providing an automated testing pipeline for neural network models

A cloud platform establishes a communication session between an agent and a user. The communication session is over an electrical medium. The cloud platform generates an interface on a client device associated with the agent. A first portion of the interface is configured to exchange messages between the agent and the user for a conversation or otherwise transcribe a conversation between the agent and the user. The cloud platform obtains, at a first time, a set of utterances from a transcript of the conversation. The cloud platform accesses a database including a plurality of articles. The cloud platform generates relevance scores between the conversation and the plurality of articles. The cloud platform then selects a subset of articles having relevance scores above a threshold value or proportion. The identified articles are presented on a second portion of the interface.

Machine-learning based document recommendation for online real-time communication system

A text interaction record including interaction text from one or more messages between a client machine and a service provider is received at a database system. A search vector including a text embedding representing the interaction text in a multi-dimensional vector space may be determined based on the interaction text via a processor at the database system. A reference interaction record including reference interaction text may be retrieved from the database system based on the search vector. The reference interaction record may include a reference vector representing the reference interaction text in the multi-dimensional vector space. Recommended reply text is determined based on the interaction text and the reference interaction text by a large language model configured to generate the recommended reply text in response to a prompt that includes one or more natural language instructions.

Database system interaction embedding and indexing for text retrieval and generation

A method is provided for generating a ranked list of candidate responders. In some embodiments, the method includes receiving a question from a user and generating a question feature vector representing an intent of the question and a first skill set inferred from the question. The method also includes for one or more candidate responders, generating a candidate feature vector representing a skill set and questions associated with the respective candidate responder; computing a reputation score based on questions and user feedback associated with the respective candidate responder; and computing, based on the question feature vector, candidate feature vector, and reputation score, a probability score representing a prediction of the quality of an answer that would be provided by the respective candidate responder if the input question were routed to the respective candidate responder. The method further includes generating a ranked list of candidate responders using the computed probability scores.

Routing of cases using unstructured input and natural language processing

A method and system for recommending articles including: receiving a customer request from the customer during the session; generating case data for a case, by an article recommender app; configuring a training set based on the subject and description data of the customer request; identifying, by an artificial intelligence (AI) app, a first pool of articles from a knowledge database; identifying by at least one query, a second pool of articles from a case article database to into a merged pool of articles; assigning, by the AI app, an implicit label to one of the first pool and the second pool of the articles; applying a model derived by the AI app based on customer behavior and a set of features related to the case to classify each article of the merged pool of articles based at least in part on the predicted relevance of the article.

Method and system for service agent assistance of article recommendations to a customer in an app session

A system and related processing methodologies for recommending a product based on a work order are described. The system receives an input case description, including a current repair item and a current work type. Historical work orders associating a plurality of products with repair items and work types are searched for a co-occurrence of the repair item matching the current repair item, and the work type matching the current work type. Upon finding a match, the product associated with the match is added to a set of candidate products for the current work order. A similarity measure between the candidate product and current work order description, a current work type category, and popularity of the candidate product is generated and then used in the generation of a probability score for the candidate product and current work order. If the probability score meets a threshold, the candidate product is recommended.

Techniques and architectures for recommending products based on work orders

A data processing system analyzes a corpus of conversation data collected at an interactive conversation service to train an intent classification model. The intent classification model generates vectors based on the corpus of conversation data. A set of intents is selected and an intent seed input for each intent of the set of intents is input into the model to generate an intent vector corresponding to each intent. Vectors based on user inputs are generated and compared to the intent vectors to determine the intent.

Intent classification system

Growing community of inventors

San Clara, CA, United States of America

Sitaram Asur