Facebook, Inc.

Meta Platforms Technologies, LLC

Facebook Technologies, LLC

Meta Platforms, Inc.

Anuj Kumar

Francislav Petrov Penov

William Crosby Presant

Benoit F Dumoulin

Rajen Subba

Wenhai Yang

Pararth Paresh Shah

Seungwhan Moon

Wei P Chen

Fuchun Peng

Baiyang Liu

Kun Han

Michael Robert Hanson

Christian Fuegen

Honglei Liu

Armen Aghajanyan

Jonathan David Christensen

Shreyan Bakshi

Kirk LaBuda

Wenxuan Li

Fadi Botros

Fan Wang

Meryem Pinar Donmez Ediz

Omer Muzaffar

Vikas Seshagiri Rao Bhardwaj

Daniel Difranco

Crystal Annette Naomi Su Hua Nakatsu

Theodore Frank Levin

Brian Moran

Sonal Gupta

Fei Sha

Shusen Liu

Ted Li

Rushin Shah

Dmitrii Serdiuk

Yongquiang Wang

In one embodiment, a method includes receiving a first input by a user from a client system associated with the user, wherein the first input is in a voice modality, analyzing the first input to generate one or more candidate hypotheses, determining one or more modalities for presenting output generated by the one or more computing systems to the user at the client system, and sending instructions to the client system for presenting one or more suggested auto-completions corresponding to one or more of the candidate hypotheses, respectively, wherein each suggested auto-completion comprises the corresponding candidate hypothesis, and wherein the one or more suggested auto-completions are presented in the one or more determined modalities.

Voice-based auto-completions and auto-responses for assistant systems

A method includes detecting a user input comprising an incomplete three-dimensional (3D) gesture performed by one or more hands of a first user by a virtual-reality (VR) headset, selecting candidate 3D gestures from pre-defined 3D gestures based on a personalized gesture-recognition model, wherein each of the candidate 3D gestures is associated with a confidence score representing a likelihood the first user intended to input the respective candidate 3D gesture, and presenting one or more suggested inputs corresponding to one or more of the candidate 3D gestures at the VR headset.

Auto-completion for gesture-input in assistant systems

In one embodiment, a method includes training a target machine-learning model iteratively by accessing training data of content objects, training an intermediate machine-learning model that outputs contextual evaluation measurements based on the training data, generating state-indications associated with the training data, wherein the state-indications comprise user-intents, system actions, and user actions, training the target machine-learning model based on the contextual evaluation measurements, the state-indications, and an action set comprising possible system actions, extracting rules based on the target machine-learning model by a sequential pattern-mining model, generating synthetic training data based on the rules, updating the training data by adding the synthetic training data to the training data, determining if a completion condition is reached for the training, and if the completion condition is reached returning the target machine-learning model, else repeating the iterative training of the target machine-learning model.

Interpretability of deep reinforcement learning models in assistant systems

In one embodiment, a method includes receiving an initial input in a first modality from a first user from a client system associated with the first user, determining one or more intents corresponding to the initial input by an intent-understanding module, generating one or more candidate continuation-inputs based on the one or more intents, where the one or more candidate continuation-inputs are in one or more candidate modalities, respectively, and wherein the candidate modalities are different from the first modality, and sending instructions for presenting one or more suggested inputs corresponding to one or more of the candidate continuation-inputs to the client system.

In one embodiment, a method includes receiving a user input comprising a natural-language utterance by an assistant xbot from a client system associated with a user, determining a semantic representation of the user input based on a structural ontology defining a labeling syntax for parsing the natural-language utterance to semantic units comprising actions, objects, and attributes, wherein the semantic representation embeds at least one object within at least one action and declares at least one attribute of the embedded object to be acted upon, sending a request based on the semantic representation to an agent for executing a task corresponding to the user input, receiving results of the executed task mapped to a structure determined by the structural ontology from the agent, and sending from the assistant xbot to the client system instructions for presenting a response based on the results of the executed task.

Semantic representations using structural ontology for assistant systems

In one embodiment, a method includes receiving a query from a user from a client system associated with the user, determining one or more initial memory slots based on the query, accessing a memory graph associated with the user which comprises a plurality of nodes and a plurality of edges connecting the nodes, and wherein one or more of the nodes correspond to one or more episodic memories of the user, respectively, and wherein each edge corresponds to a relationship between the connected nodes, selecting one or more candidate nodes from the memory graph by one or more machine-learning models based on the initial memory slots, generating a response based on the initial memory slots and episodic memories corresponding to the selected candidate nodes, and sending instructions for presenting the response to the client system in response to the query.

Memory grounded conversational reasoning and question answering for assistant systems

In one embodiment, a method includes receiving a query from a user from a client system associated with the user, accessing a knowledge graph comprising a plurality of nodes and edges connecting the nodes, wherein each node corresponds to an entity and each edge corresponds to a relationship between the entities corresponding to the connected nodes, determining one or more initial entities associated with the query based on the query, selecting one or more candidate nodes by a conversational reasoning model from the knowledge graph corresponding to one or more candidate entities, respectively, wherein each candidate node is selected based on the nodes corresponding to the initial entities, dialog states associated with the query, and a context associated with the query, generating a response based on the initial entities and the candidate entities, and sending instructions for presenting the response to the client system in response to the query.

Conversational reasoning with knowledge graph paths for assistant systems

In one embodiment, a method includes detecting a user input comprising an incomplete gesture performed by one or more hands of a first user by a client system associated with the first user; selecting one or more candidate gestures from a plurality of pre-defined gestures by the client system based on a personalized gesture-recognition model, wherein each of the candidate gestures is associated with a confidence score representing a likelihood the first user intended to input the respective candidate gesture, and presenting one or more suggested inputs corresponding to one or more of the candidate gestures at the client system.

Exemplary embodiments relate to improvements in spoken language understanding (SLU) systems. Conventionally, SLU systems include an automatic speech recognition (ASR) component configured to receive an input of audio data and to generate a textual representation of the audio data. Conventional SLU systems also include a natural language understanding (NLU) component configured to receive a text-based transcript and perform language-based tasks such as domain classification, intent determination, and slot-filling. However, these two components are typically trained separately based on different metrics. In real-world situations, errors in the ASR component propagate to the NLU component, which degrades the performance of the overall system. Exemplary embodiments described herein perform SLU in an end-to-end manner that infers semantic meaning directly from audio features without an intermediate text representation. This may allow for more a more accurate translation performed in a more resource-efficient manner (particularly in terms of processing resources).

Methods and systems for performing end-to-end spoken language analysis

Auto-completion for multi-modal user input in assistant systems

Techniques for database versioning are described. In one embodiment, an apparatus may comprise a database change management component operative to compare a developer table to a reference table to determine a database change set, wherein both the developer table and the reference table are based on a target table; a database conflict management component operative to compare the database change set to the target table to determine a conflicting change set; and a user interface component operative to display the conflicting change set where the conflicting change set comprises one or more conflicting changes; and indicate a conflict-free change set where the conflicting change set is empty. Other embodiments are described and claimed.

Techniques for database versioning

In one embodiment, a method includes receiving a user request for a summarization of a particular type of content objects from a client system associated with a first user, determining one or more modalities associated with the user request, selecting a plurality of content objects of the particular type based on a user profile of the first user, wherein the user profile comprises one or more confidence scores associated with one or more subjects associated with the first user, respectively, and wherein the plurality of content objects are selected based on the one or more confidence scores, generating a summary of each content object based on the user profile and the determined modalities, and sending, to the client system in response to the user request, instructions for presenting the summaries of the plurality of content objects, wherein the summaries are presented via one or more of the determined modalities.

Generating personalized content summaries for users

A user interacts with a virtual digital assistant with the intent that it provides assistance with a task. The user sends messages to the virtual digital assistant that include content obtained via user input at a client device. An intent determination model is applied to the content to identify the user's intent. The virtual digital assistant identifies agents that are capable of servicing the intent are identified and retrieves contextual data relating to the message from a data store. An intent arbitration model is used to select one of the agents which is activated to provide assistance with the task. The contextual information may include global metrics of agent performance and/or information regarding the user's preferences.

Intent arbitration for a virtual assistant

In one embodiment, a method includes receiving a user input from a client system associated with a first user, wherein the user input comprises an incomplete gesture performed by the first user, calculating one or more confidence scores by an intent-understanding module for one or more intents corresponding to the incomplete gesture, determining the calculated confidence scores associated with each of the intents are below a threshold score, selecting candidate gestures from a plurality of pre-defined gestures based on a personalized gesture-recognition model responsive to determining that the calculated confidence scores for each of the intents are below the threshold score, wherein each of the candidate gestures is associated with a confidence score representing a likelihood the first user intended to input the respective candidate gesture, and sending instructions for presenting one or more suggested inputs corresponding to one or more of the candidate gestures to the client system.

Systems, methods, and non-transitory computer-readable media can receive, from a first entity, training data for training an intent model associated with a first intent of a plurality of intents. A first intent model associated with the first intent is generated based on the training data. The first intent model is made available in an intent marketplace for access by a second entity.

Growing community of inventors

Santa Clara, CA, United States of America

Anuj Kumar