Meta Platforms Technologies, LLC

Facebook Technologies, LLC

Meta Platforms, Inc.

Paul Anthony Crook

Rajen Subba

Xiaohu Liu

Francislav Petrov Penov

Seungwhan Moon

Shivani Poddar

Baiyang Liu

Zhiguang Wang

Pararth Paresh Shah

Bing Liu

Kai Jun Weng

Krishna Mittal

Shubham Khandelwal

Eun Joon Cho

Xin Ming Fan

In one embodiment, a method includes receiving a user request from a first user from a client system associated with a first user, wherein the user request comprise a gesture-input from the first user and a speech-input from the first user, determining an intent corresponding to the user request based on the gesture-input by a personalized gesture-classification model associated with the first user, executing one or more tasks based on the determined intent and the speech-input, and sending instructions for presenting execution results of the one or more tasks to the client system responsive the user request.

Personalized gesture recognition for user interaction with assistant systems

In one embodiment, a method includes maintaining contextual information from a first dialog session, wherein the contextual information comprises identifiers of entities and slots, receiving a user request during the first dialog session, determining a context carryover is required for an agent to execute a task associated with the user request, determining a first subset of the identifiers of the entities and a second subset of the slots of the contextual information from the first dialog session are shareable with the agent based on the agent and first resources of the identifiers of the entities and second resources of the slots, and executing the task by the agent based on the first subset of the identifiers of the entities and the second subset of the slots of the shareable contextual information.

Context carryover across tasks for assistant systems

In one embodiment, a method includes maintaining contextual information from a first user request associated with a first task by a context engine, wherein the first task is associated with a first agent, receiving a second user request associated with a second task from a client system, wherein the second user request comprises an ambiguous mention and the second task is associated with a second agent, determining a context carryover is required for the second agent to execute the second task, determining the ambiguous mention corresponds to one or more data items associated with the contextual information from the first user request, and providing the one or more data items to the second agent for execution of the second task.

In one embodiment, a method includes receiving, at a client system, an audio input, where the audio input comprises a coreference to a target object, accessing visual data from one or more camera associated with the client system, where the visual data comprises images portraying one or more objects, resolving the coreference to the target object from among the one or more objects, resoling the target object to a specific entity, and providing, at the client system, a response to the audio input, where the response comprises information about the specific entity.

Multimodal entity and coreference resolution for assistant systems

In one embodiment, a method includes receiving a user request from a client system associated with a user, determining that the user request corresponds to a first suspended task, retrieving a first dialog state of the first suspended task from a dialog history associated with the user, generating a summary of the first suspended task based on the first dialog state using a natural-language generating (NLG) module, and sending instructions to the client system for providing the summary of the first suspended task to the user.

Conversation summarization during user-control task execution for assistant systems

In one embodiment, a method includes receiving, from a client system associated with a user, a user request comprising a reference to a target object, accessing visual data from the client system, wherein the visual data comprises images portraying the target object and one or more additional objects, and wherein attribute information of the target object is recorded in a multimodal dialog state, resolving the reference to the target object based on the attribute information recorded in the multimodal dialog state, determining relational information between the target object and one or more of the additional objects portrayed in the visual data, and sending, to the client system, instructions for presenting a response to the user request, wherein the response comprises the attribute information and the determined relational information.

Multimodal dialog state tracking and action prediction for assistant systems

In one embodiment, a method includes receiving a user input at a client system, wherein the user input is associated with one or more intents and one or more slots, generating one or more first dialog acts based on the user input, calculating a task-confidence score based on one or more intent-confidence scores associated with the one or more intents, respectively, and one or more slot-confidence scores associated with the one or more slots, respectively, generating one or more second dialog acts modifying the one or more first dialog acts responsive to the task-confidence score being less than a threshold score, and presenting a response to the user input at the client system, wherein the response is based on one or more of the first dialog acts or the second dialog acts.

Dialog session override policies for assistant systems

In one embodiment, a method includes accessing visual data from a client system associated with a user, wherein the visual data comprises images portraying one or more objects, receiving, from the client system, a user request, wherein the user request comprises a coreference to a target object, resolving the coreference to the target object from among the one or more objects, resolving the target object to a specific entity, and sending, to the client system, instructions for providing a response to the user request, wherein the response comprises attribute information about the specific entity.

In one embodiment, a method includes, by one or more computing systems, receiving, from a client system, an input in a multi-turn message thread, parsing the input to identify a plurality of hypothesis dialog states associated with the input, generating a plurality of functions corresponding to the plurality of hypothesis dialog states, calculating a plurality of probability scores for the plurality of functions, respectively, based on a prior dialog state associated with the message thread and a current user context, wherein each probability score represents a likelihood that the respective function is associated with an intended dialog state, and storing the plurality of functions if each of the probability scores is below a threshold probability score, wherein the plurality of functions is stored as a hierarchical set of functions that is used as the prior dialog state in response to a next input in the message thread.

Dialog state tracking for assistant systems

In one embodiment, a method includes receiving a user input from a client system associated with a user, determining a task based on the user input and a confidence score associated with the task, generating one or more first dialog acts based on a task policy which specifies dialog acts associated with the task, generating one or more second dialog acts based on an override policy responsive to the confidence score being less than a threshold score, wherein the override policy specifies dialog acts that modify dialog acts specified by the task policy; and sending instructions for presenting a response to the user input to the client system, wherein the response is based on one or more of the first dialog acts or the second dialog acts.

In one embodiment, a method includes accessing a plurality of input tuples associated with a first user from a data store, wherein each input tuple comprises a gesture-input and a corresponding speech-input, determining a plurality of intents corresponding to the plurality of speech-inputs, respectively, by a natural-language understanding (NLU) module, generating a plurality of feature representations for the plurality of gesture-inputs based on one or more machine-learning models, determining a plurality of gesture identifiers for the plurality of gesture-inputs, respectively, based on their respective feature representations, associating the plurality of intents with the plurality of gesture identifiers, respectively, and training a personalized gesture-classification model for the first user based on the plurality of feature representations of their respective gesture-inputs and the associations between the plurality of intents and their respective gesture identifiers.

Growing community of inventors

Newcastle, WA, United States of America

Paul Anthony Crook