Salesforce, Inc.

Salesforce Com, Inc.

Seagate Technology Incorporated

Zachary Alexander

Na Cheng

Sitaram Asur

Anuprit Kale

Scott Thurston Rickard, Jr

Yixin Mao

Caiming Xiong

Jayesh Govindarajan

Shashank Harinath

Weiping Peng

Chien-Sheng Wu

Jacob Nathaniel Huffman

Shiva Kumar Pentyala

Edgar Gerardo Velasco

Aron Kale

Clifford Z Huang

Naren M Chittar

Wenhao Liu

Nicholas Beng Tek Geh

Peter Francis White

Regunathan Radhakrishnan

Prafulla Kumar Choubey

Ammar Haris

Feifei Jiang

Victor Winslow Yee

Rick Lindstrom

Alampallam R Ramachandran

Tiffany Deiandra McKenzie

Yuanxin Wang

Victor Yee

Tracy Morgan Backes

Andrei Khurshudov

Tian Xie

Wojciech Kryscinski

Christian Bruun Madsen

Zineb Laraki

Alexander R Fabbri

Khoa Le

Siddharth Rajaram

J Justin Donaldson

Avanthika Ramesh

Elizabeth Balsam

Shubham Mehrotra

Greg Thomas Pascale

Joseph R Zeimen

Nicholas Feinig

Sameer Abhinkar

Fermin Ordaz

Colleen Smith

Vishal Shah

Alejandro Gabriel Perez Rodriguez

Database systems and methods are provided for assigning structural metadata to records and creating automations using the structural metadata. One method of assisting creation of an automation for conversational interactions involves providing a first graphical user interface (GUI) display including graphical indicia of a plurality of semantic groups associated with historical conversations, in response to selection of a semantic group, providing a second GUI display including second graphical indicia of a plurality of cluster groups of conversations associated with the selected semantic group, in response to second selection of a cluster group, providing a third GUI display including third graphical indicia of representative utterances associated with respective conversations of a subset of historical conversations assigned to the selected cluster group, and in response to third selection of a GUI element on the third GUI display, providing a fourth GUI display including GUI elements for defining the automation.

Database systems and methods of defining conversation automations

Database systems and methods are provided for assigning structural metadata to records and creating automations using the structural metadata. One method of assigning structural metadata to a group of conversation records involves receiving a user input modification pertaining to a group of semantically similar conversations, automatically reassigning a conversation to a different group of semantically similar conversations based on its representative utterance in a manner that is influenced by the user input modification, and automatically updating, at a database system, a record associated with the conversation to include metadata identifying the different group of semantically similar conversations.

Database systems with user-configurable automated metadata assignment

Embodiments described herein provide a training framework for generative NLP models. Specifically, the training input, e.g., in the form of a sequence of tokens representing a user-agent dialogue, may be randomly masked for a few spans, which can be one or more tokens, one or more words, one or more sentences, or one or more paragraphs. These masked spans are replaced with their embeddings generated from pre-trained large language models are then used for training the NLP model.

Systems and methods for training a neural network model using knowledge from pre-trained large language models

The system receives a base machine learning model trained using a generic dataset. For example, the base machine learning model may be an off-the-shelf machine learning based model. The base machine learning model is trained to receive an input and generate a feature vector representing the input. The input may be a natural language expression, an image, or any other type of input. The system receives a domain specific training dataset based on known categories for input values. The system determines an orthogonal transformation for reducing the dimensions of the base machine learning model using on the domain specific training dataset. The system applies the orthogonal transformation to the base machine learning model to obtain a domain specific machine learning model. The system uses the domain specific machine learning model for processing inputs, for example, in a production environment.

Domain adaptation of machine learning models for classification

Embodiments described herein provide a training framework for generative NLP models that operate on previously learnt knowledge from pretrained large language models. Specifically, to train an NLP model to generate a response to a user utterance (e.g., 'resolve login issue'), document embeddings of support IT documents encoded by a pretrained LLM are fed to an NLP decoder together with a training dialogue (e.g., a dialogue between the chat agent on how to “resolve login issue”). The NLP decoder can thus be trained by a causal language modeling loss computed based on the predicted next token and the ground-truth token from the training dialogue.

Embodiments described herein provide a structured conversation summarization framework. A user interface may be provided which allows an agent to perform a conversation with a customer, for example regarding resolving a customer support issue. Utterances by both the agent and customer may be stored, and at the end of the conversation, the utterances may be used to generate a structured summary. The structured summary may include components such as a general summary, an issue summary, and a resolution summary. Using neural network models and heuristics, each component of the summary may be automatically generated.

Systems and methods for structured conversation summarization

Embodiments described herein provide systems and methods for document recommendation. A system receives a set of training data including a plurality of documents. The system determines whether the set of training data includes annotated contextual information corresponding to the plurality of documents. The system trains supervised and/or unsupervised models based on the availability of data. The models are used to generate vectors representing the documents. During a live text conversation, text from the conversation may be vectorized using the models and the vectors compared to those representing the documents in order to find the most relevant documents. The system may generate an indication of a recommended document.

Systems and methods for real-time document recommendation

Described herein are systems, apparatus, methods and computer program products for machine learning intent classification. In various embodiments, historical utterances provided by users may be utilized for bot training. Context and personally identifiable information may be removed from the utterances. The utterances may be associated with vectors. The utterances and vectors may be used to determine recommendations.

Secure complete phrase utterance recommendation system

Database systems and methods are provided for assigning structural metadata to records and creating automations using the structural metadata. One method of assigning structural metadata to a record associated with a conversation involves obtaining a plurality of utterances associated with the conversation, identifying, from among the plurality of utterances, a representative utterance for semantic content of the conversation, assigning the conversation to a group of semantically similar conversations based on the representative utterance, and automatically updating the record associated with the conversation at a database system to include metadata identifying the group of semantically similar conversations.

Database systems with automated structural metadata assignment

Embodiments described herein provide an automated testing pipeline for providing a testing dataset for testing a trained neural network model trained using a first training dataset. A first testing dataset for the trained neural network including a first plurality of user queries is received. A dependency parser is used to filter the first plurality of user queries based on one or more action verbs. A pretrained language model is used to rank the remaining user queries based on respective relationships with queries in the first training dataset. Further, user queries that are classified as keyword matches with the queries in the first training dataset using a bag of words classifier are removed. A second testing dataset is generated using the ranked remaining user queries. Testing outputs are generated, by the trained neural network model, using the second testing dataset.

Systems and methods for providing an automated testing pipeline for neural network models

A cloud platform establishes a communication session between an agent and a user. The communication session is over an electrical medium. The cloud platform generates an interface on a client device associated with the agent. A first portion of the interface is configured to exchange messages between the agent and the user for a conversation or otherwise transcribe a conversation between the agent and the user. The cloud platform obtains, at a first time, a set of utterances from a transcript of the conversation. The cloud platform accesses a database including a plurality of articles. The cloud platform generates relevance scores between the conversation and the plurality of articles. The cloud platform then selects a subset of articles having relevance scores above a threshold value or proportion. The identified articles are presented on a second portion of the interface.

Machine-learning based document recommendation for online real-time communication system

A text interaction record including interaction text from one or more messages between a client machine and a service provider is received at a database system. A search vector including a text embedding representing the interaction text in a multi-dimensional vector space may be determined based on the interaction text via a processor at the database system. A reference interaction record including reference interaction text may be retrieved from the database system based on the search vector. The reference interaction record may include a reference vector representing the reference interaction text in the multi-dimensional vector space. Recommended reply text is determined based on the interaction text and the reference interaction text by a large language model configured to generate the recommended reply text in response to a prompt that includes one or more natural language instructions.

Database system interaction embedding and indexing for text retrieval and generation

Database systems and methods are provided for assigning structural metadata to records and creating automations using the structural metadata. One method of assigning structural metadata involves determining a candidate group of semantically similar conversations based on unassigned conversations, determining a clustering performance metric associated with the candidate group based on a relationship between the candidate group and a plurality of existing groups of semantically similar conversations, and when the clustering performance metric is greater than a threshold, automatically assigning one or more unassigned conversations to the candidate group based on the representative utterances associated therewith and automatically updating one or more associated records at a database system to include metadata identifying the candidate group.

Database systems with adaptive automated metadata assignment

Disclosed are some implementations of systems, apparatus, methods and computer program products for integrating question generation and answer retrieval in a question answer system. The system generates a question using a set of documents and determines whether it is semantically distinct from questions in a question-answer repository. After determining that the question is semantically distinct from questions in the question-answer repository, the system adds the question to the question-answer repository. Upon receipt of a user-submitted question, the system uses the question-answer repository to identify a semantically similar question. The system retrieves an answer corresponding to the identified question from the question-answer repository and provides the answer in response to the user-submitted question.

Question answering using dynamic question-answer database

Growing community of inventors

Snoqualmie, WA, United States of America

Zachary Alexander