Salesforce, Inc.

Salesforce Com, Inc.

Na Cheng

Zachary Alexander

Sitaram Asur

Anuprit Kale

Weiping Peng

Edgar Gerardo Velasco

Jayesh Govindarajan

Jacob Nathaniel Huffman

Peter Francis White

Caiming Xiong

Shashank Harinath

Yixin Mao

Feifei Jiang

Khoa Le

Rick Lindstrom

Victor Yee

Victor Winslow Yee

Prithvi Krishnan Padmanabhan

Wenhao Liu

Chien-Sheng Wu

Nicholas Beng Tek Geh

Wojciech Kryscinski

Shiva Kumar Pentyala

Anjan Goswami

Pingping Xiu

Alexander R Fabbri

Aditya Sakhuja

Son Thanh Chang

Aron Kale

Gary Brandeleer

Shubham Mehrotra

Shilpa Bhagavath

Abhishek Sharma

Kavya Murali

Nicholas Feinig

Sameer Abhinkar

Fermin Ordaz

Zhichao Wang

Bin Bi

Yuanxin Wang

Joseph R Zeimen

Suhas Satish

Ziwei Chen

Alejandro Gabriel Perez Rodriguez

Vishal Shah

Colleen Smith

Christopher Larry

Nandini Suresh Kumar

An application server may receive, from a client and at an interface for accessing a large language model, a prompt for a response from the large language model. The application server may receive, via a model interface, a streaming output of the large language model, where the streaming output includes a first portion of the response and a threshold number of tokens. The application server may then provide the first portion of the response to a scoring model that determines a first incremental score indicating a first probability that the first portion of the response includes content from one or more content categories. The application server may transmit, to the client and based on the first probability, the first portion of the response, an indication of the first incremental score, or both.

Generative responses with trust for large language models

In some systems, a set of sentences of a relatively large document may be vectorized into a set of vectors via an embedding model for summarization. Further, a subset of vectors of the set of vectors may be selected via a farthest point sampling (FPS) procedure based on a vector-space distance between respective vectors of the subset of vectors. Moreover, the subset of vectors that are associated with a subset of sentences may be ordered based on the order of the subset of sentences within the set of sentences of the document. Further, to generate a summary of the document, a query may be transmitted to a large language model (LLM) that includes a summarization prompt and the subset of sentences that correspond with the selected subset of vectors. A summary of the document may then be received from the LLM based on transmitting the query.

Extractive-abstractive large language model summarization with farthest point sampling

Embodiments described herein provide a structured conversation summarization framework. A user interface may be provided which allows an agent to perform a conversation with a customer, for example regarding resolving a customer support issue. Utterances by both the agent and customer may be stored, and at the end of the conversation, the utterances may be used to generate a structured summary. The structured summary may include components such as a general summary, an issue summary, and a resolution summary. Using neural network models and heuristics, each component of the summary may be automatically generated.

Systems and methods for structured conversation summarization

Embodiments described herein provide systems and methods for document recommendation. A system receives a set of training data including a plurality of documents. The system determines whether the set of training data includes annotated contextual information corresponding to the plurality of documents. The system trains supervised and/or unsupervised models based on the availability of data. The models are used to generate vectors representing the documents. During a live text conversation, text from the conversation may be vectorized using the models and the vectors compared to those representing the documents in order to find the most relevant documents. The system may generate an indication of a recommended document.

Systems and methods for real-time document recommendation

Described herein are systems, apparatus, methods and computer program products for machine learning intent classification. In various embodiments, historical utterances provided by users may be utilized for bot training. Context and personally identifiable information may be removed from the utterances. The utterances may be associated with vectors. The utterances and vectors may be used to determine recommendations.

Secure complete phrase utterance recommendation system

A method that includes receiving an input at an interactive conversation service that uses an intent classification model. The method may further include generating, using an encoder model of the intent classification model, a set of output vectors corresponding to the input, where the encoder model is configured to determine a set of metrics corresponding to intent classifications. The method may further include determining, using an outlier detection model of the intent classification model, whether the input is in-domain or out-of-domain (OOD) based on a first vector of the set of output vectors satisfying a domain threshold relative to one or more of the intent classifications. The method may further include outputting, by the intent classification model, a second vector of the set of output vectors that indicates the set of metrics corresponding to the intent classifications or an indication that the input is OOD.

Multi-lingual intent model with out-of-domain detection

Disclosed are some implementations of systems, apparatus, methods and computer program products for integrating question generation and answer retrieval in a question answer system. The system generates a question using a set of documents and determines whether it is semantically distinct from questions in a question-answer repository. After determining that the question is semantically distinct from questions in the question-answer repository, the system adds the question to the question-answer repository. Upon receipt of a user-submitted question, the system uses the question-answer repository to identify a semantically similar question. The system retrieves an answer corresponding to the identified question from the question-answer repository and provides the answer in response to the user-submitted question.

Question answering using dynamic question-answer database

A system uses conversation engines to process natural language requests and conduct automatic conversations with users. The system generates responses to users in an online conversation. The system ranks generated user responses for the online conversation. The system generates a context vector based on a sequence of utterances of the conversation and generates response vectors for generated user responses. The system ranks the user responses based on a comparison of the context vectors and user response vectors. The system uses a machine learning based model that uses a pretrained neural network that supports multiple languages. The system determines a context of an utterance based on utterances in the conversation. The system generates responses and ranks them based on the context. The ranked responses are used to respond to the user.

Machine learning based models for automatic conversations in online systems

A method and system for recommending articles including: receiving a customer request from the customer during the session; generating case data for a case, by an article recommender app; configuring a training set based on the subject and description data of the customer request; identifying, by an artificial intelligence (AI) app, a first pool of articles from a knowledge database; identifying by at least one query, a second pool of articles from a case article database to into a merged pool of articles; assigning, by the AI app, an implicit label to one of the first pool and the second pool of the articles; applying a model derived by the AI app based on customer behavior and a set of features related to the case to classify each article of the merged pool of articles based at least in part on the predicted relevance of the article.

Method and system for service agent assistance of article recommendations to a customer in an app session

A system and related processing methodologies for recommending a product based on a work order are described. The system receives an input case description, including a current repair item and a current work type. Historical work orders associating a plurality of products with repair items and work types are searched for a co-occurrence of the repair item matching the current repair item, and the work type matching the current work type. Upon finding a match, the product associated with the match is added to a set of candidate products for the current work order. A similarity measure between the candidate product and current work order description, a current work type category, and popularity of the candidate product is generated and then used in the generation of a probability score for the candidate product and current work order. If the probability score meets a threshold, the candidate product is recommended.

Techniques and architectures for recommending products based on work orders

Disclosed are some implementations of systems, apparatus, methods and computer program products for extracting topics from a corpus of exchanges. The system generates vector representations of utterances of an entity common to the exchanges and uses the vector representations to cluster the utterances. The system labels the clusters and uses the labeled clusters to generate an exchange label sequence for each of the exchanges, where each exchange label sequence corresponds to a sequence of utterances generated by the entity. The system processes the exchange label sequences to generate one or more subsets of the utterances, where each of the subsets corresponds to a particular topic.

Unsupervised dialogue topic extraction

A system is provided for a machine learning engine using clustered case objects in a case management system. The system includes a multi-layer neural network. The system is configured to receive case object data comprising a case object and contextual objects in the case management system associated with the case object, the contextual objects comprising word vectors, generate a context embedding for the case object using the word vectors for the contextual objects, and cluster the case object with other case objects in the case management system based on the context embedding for the case object and other context embeddings for the other case objects.

Case object context embeddings for machine learning training of case context

A system is configured to analyze a corpus of historical chat data to identify the list of 'best' responses. As such, the user is not required to identify a list of canned responses for input into the system. The described system uses a context word embedding function and response word embedding function to generate context vectors and response vectors corresponding to the corpus of conversation data, and the vectors are represented by a respective context matrix and a response matrix. The system processes these matrices to generate scores for responses, clusters the responses, and identifies the responses corresponding to the best scores for each cluster.

Identification of response list

Computing systems, database systems, and related methods are provided for recommending values for fields of database objects and dynamically updating a recommended value for a field of a database record in response to updated auxiliary data associated with the database record. One method involves obtaining associated conversational data, segmenting the conversational data, converting each respective segment of conversational data into a numerical representation, generating a combined numerical representation of the conversational data based on the sequence of numerical representations using an aggregation model, generating the recommended value based on the combined numerical representation of the conversational data using a prediction model associated with the field, and autopopulating the field of the case database object with the recommended value.

Dynamic field value recommendation methods and systems

Growing community of inventors

Dublin, CA, United States of America

Na Cheng