Salesforce, Inc.

Salesforce Com, Inc.

Seagate Technology Incorporated

Zachary Alexander

Na Cheng

Sitaram Asur

Anuprit Kale

Scott Thurston Rickard, Jr

Jayesh Govindarajan

Yixin Mao

Caiming Xiong

Weiping Peng

Shashank Harinath

Jacob Nathaniel Huffman

Edgar Gerardo Velasco

Aron Kale

Clifford Z Huang

Naren M Chittar

Wenhao Liu

Nicholas Beng Tek Geh

Chien-Sheng Wu

Peter Francis White

Shiva Kumar Pentyala

Regunathan Radhakrishnan

Ammar Haris

Feifei Jiang

Victor Winslow Yee

Rick Lindstrom

Alampallam R Ramachandran

Tiffany Deiandra McKenzie

Yuanxin Wang

Victor Yee

Tracy Morgan Backes

Andrei Khurshudov

Tian Xie

Wojciech Kryscinski

Christian Bruun Madsen

Alexander R Fabbri

Prafulla Kumar Choubey

Khoa Le

Siddharth Rajaram

J Justin Donaldson

Elizabeth Balsam

Shubham Mehrotra

Greg Thomas Pascale

Joseph R Zeimen

Sameer Abhinkar

Nicholas Feinig

Fermin Ordaz

Colleen Smith

Vishal Shah

Alejandro Gabriel Perez Rodriguez

The system receives a base machine learning model trained using a generic dataset. For example, the base machine learning model may be an off-the-shelf machine learning based model. The base machine learning model is trained to receive an input and generate a feature vector representing the input. The input may be a natural language expression, an image, or any other type of input. The system receives a domain specific training dataset based on known categories for input values. The system determines an orthogonal transformation for reducing the dimensions of the base machine learning model using on the domain specific training dataset. The system applies the orthogonal transformation to the base machine learning model to obtain a domain specific machine learning model. The system uses the domain specific machine learning model for processing inputs, for example, in a production environment.

Domain adaptation of machine learning models for classification

Embodiments described herein provide a training framework for generative NLP models that operate on previously learnt knowledge from pretrained large language models. Specifically, to train an NLP model to generate a response to a user utterance (e.g., 'resolve login issue'), document embeddings of support IT documents encoded by a pretrained LLM are fed to an NLP decoder together with a training dialogue (e.g., a dialogue between the chat agent on how to “resolve login issue”). The NLP decoder can thus be trained by a causal language modeling loss computed based on the predicted next token and the ground-truth token from the training dialogue.

Systems and methods for training a neural network model using knowledge from pre-trained large language models

Embodiments described herein provide a structured conversation summarization framework. A user interface may be provided which allows an agent to perform a conversation with a customer, for example regarding resolving a customer support issue. Utterances by both the agent and customer may be stored, and at the end of the conversation, the utterances may be used to generate a structured summary. The structured summary may include components such as a general summary, an issue summary, and a resolution summary. Using neural network models and heuristics, each component of the summary may be automatically generated.

Systems and methods for structured conversation summarization

Embodiments described herein provide systems and methods for document recommendation. A system receives a set of training data including a plurality of documents. The system determines whether the set of training data includes annotated contextual information corresponding to the plurality of documents. The system trains supervised and/or unsupervised models based on the availability of data. The models are used to generate vectors representing the documents. During a live text conversation, text from the conversation may be vectorized using the models and the vectors compared to those representing the documents in order to find the most relevant documents. The system may generate an indication of a recommended document.

Systems and methods for real-time document recommendation

Described herein are systems, apparatus, methods and computer program products for machine learning intent classification. In various embodiments, historical utterances provided by users may be utilized for bot training. Context and personally identifiable information may be removed from the utterances. The utterances may be associated with vectors. The utterances and vectors may be used to determine recommendations.

Secure complete phrase utterance recommendation system

Database systems and methods are provided for assigning structural metadata to records and creating automations using the structural metadata. One method of assigning structural metadata to a record associated with a conversation involves obtaining a plurality of utterances associated with the conversation, identifying, from among the plurality of utterances, a representative utterance for semantic content of the conversation, assigning the conversation to a group of semantically similar conversations based on the representative utterance, and automatically updating the record associated with the conversation at a database system to include metadata identifying the group of semantically similar conversations.

Database systems with automated structural metadata assignment

Embodiments described herein provide an automated testing pipeline for providing a testing dataset for testing a trained neural network model trained using a first training dataset. A first testing dataset for the trained neural network including a first plurality of user queries is received. A dependency parser is used to filter the first plurality of user queries based on one or more action verbs. A pretrained language model is used to rank the remaining user queries based on respective relationships with queries in the first training dataset. Further, user queries that are classified as keyword matches with the queries in the first training dataset using a bag of words classifier are removed. A second testing dataset is generated using the ranked remaining user queries. Testing outputs are generated, by the trained neural network model, using the second testing dataset.

Systems and methods for providing an automated testing pipeline for neural network models

A cloud platform establishes a communication session between an agent and a user. The communication session is over an electrical medium. The cloud platform generates an interface on a client device associated with the agent. A first portion of the interface is configured to exchange messages between the agent and the user for a conversation or otherwise transcribe a conversation between the agent and the user. The cloud platform obtains, at a first time, a set of utterances from a transcript of the conversation. The cloud platform accesses a database including a plurality of articles. The cloud platform generates relevance scores between the conversation and the plurality of articles. The cloud platform then selects a subset of articles having relevance scores above a threshold value or proportion. The identified articles are presented on a second portion of the interface.

Machine-learning based document recommendation for online real-time communication system

A text interaction record including interaction text from one or more messages between a client machine and a service provider is received at a database system. A search vector including a text embedding representing the interaction text in a multi-dimensional vector space may be determined based on the interaction text via a processor at the database system. A reference interaction record including reference interaction text may be retrieved from the database system based on the search vector. The reference interaction record may include a reference vector representing the reference interaction text in the multi-dimensional vector space. Recommended reply text is determined based on the interaction text and the reference interaction text by a large language model configured to generate the recommended reply text in response to a prompt that includes one or more natural language instructions.

Database system interaction embedding and indexing for text retrieval and generation

Database systems and methods are provided for assigning structural metadata to records and creating automations using the structural metadata. One method of assigning structural metadata involves determining a candidate group of semantically similar conversations based on unassigned conversations, determining a clustering performance metric associated with the candidate group based on a relationship between the candidate group and a plurality of existing groups of semantically similar conversations, and when the clustering performance metric is greater than a threshold, automatically assigning one or more unassigned conversations to the candidate group based on the representative utterances associated therewith and automatically updating one or more associated records at a database system to include metadata identifying the candidate group.

Database systems with adaptive automated metadata assignment

Disclosed are some implementations of systems, apparatus, methods and computer program products for integrating question generation and answer retrieval in a question answer system. The system generates a question using a set of documents and determines whether it is semantically distinct from questions in a question-answer repository. After determining that the question is semantically distinct from questions in the question-answer repository, the system adds the question to the question-answer repository. Upon receipt of a user-submitted question, the system uses the question-answer repository to identify a semantically similar question. The system retrieves an answer corresponding to the identified question from the question-answer repository and provides the answer in response to the user-submitted question.

Question answering using dynamic question-answer database

A system uses conversation engines to process natural language requests and conduct automatic conversations with users. The system generates responses to users in an online conversation. The system ranks generated user responses for the online conversation. The system generates a context vector based on a sequence of utterances of the conversation and generates response vectors for generated user responses. The system ranks the user responses based on a comparison of the context vectors and user response vectors. The system uses a machine learning based model that uses a pretrained neural network that supports multiple languages. The system determines a context of an utterance based on utterances in the conversation. The system generates responses and ranks them based on the context. The ranked responses are used to respond to the user.

Machine learning based models for automatic conversations in online systems

Methods and systems for answering frequently asked questions are described. An utterance is received. A decision score that is indicative of the likelihood that the utterance is answerable according to a set of frequently asked questions and associated answers is determined for the utterance. A candidate answer from the associated answers and a selection score for the candidate answer are determined for the utterance. A total score for the candidate answer is determined based on the decision score and the selection score. The total score is indicative of the likelihood that the candidate answer is a correct answer for the utterance according to the set of frequently asked questions and associated answers.

Methods and systems of answering frequently asked questions (FAQs)

An online system stores objects representing potential transactions of an enterprise. The online system uses predictor models to determine an aggregate score based on values of the objects associated with a time interval, for example, a month. Each object is configured to take one of a plurality of states. The online system stores historical data describing activities associated with potential transaction objects and uses the stored data for generating the predictor models. The online system categorizes the objects into bins based on states of the objects. The online system may generate different predictions for each category. The online system may use machine learning based models as predictor models. The online system extracts features describing potential transaction objects and provides these as input to the predictor model.

Predicting aggregate value of objects representing potential transactions based on potential transactions expected to be created

Growing community of inventors

Snoqualmie, WA, United States of America

Zachary Alexander