Facebook, Inc.

Meta Platforms, Inc.

Meta Platforms Technologies, LLC

Facebook Technologies, LLC

Rajen Subba

Xiaohu Liu

Benoit F Dumoulin

Paul Anthony Crook

Baiyang Liu

Seungwhan Moon

Francislav Petrov Penov

Michael Robert Hanson

Emmanouil Koukoumidis

Jinsong Yu

Antoine Roland Raux

Anuj Kumar

Wenhai Yang

Rushin N Shah

Heidi Young

Pararth Paresh Shah

Chandrasekhar Iyer

Jeremy Gillmor Kahn

Shivani Poddar

Fuchun Peng

Mohsen M Agsen

Shawn C P Mei

Savas Parastatidis

Zhiguang Wang

Anusha Balakrishnan

Ashwini Challa

Jiedan Zhu

Stefan Nelson-Lindall

Scott Martin

Honglei Liu

Jian Yang

Bing Liu

Xiaolei Li

Hao Zhou

Yulong Qiu

Daniel Chai

Hu Xu

Kartikeya Upasani

In one embodiment, a method includes receiving a user request from a first user from a client system associated with a first user, wherein the user request comprise a gesture-input from the first user and a speech-input from the first user, determining an intent corresponding to the user request based on the gesture-input by a personalized gesture-classification model associated with the first user, executing one or more tasks based on the determined intent and the speech-input, and sending instructions for presenting execution results of the one or more tasks to the client system responsive the user request.

Personalized gesture recognition for user interaction with assistant systems

In one embodiment, a method includes receiving a user query inputted on a head-mounted device from the head-mounted device, wherein the user query corresponds to multiple dialog-intents, executing multiple tasks corresponding to the multiple dialog-intents, generating a multi-perspective response by a stitching model based on two or more of execution results of the multiple tasks, wherein the stitching model combines the two or more of the execution results based on natural language processing, and wherein the multi-perspective response comprises a natural-language response combining the two or more execution results, and sending instructions to the head-mounted device for presenting the multi-perspective response on the head-mounted device.

Generating multi-perspective responses by assistant systems

In one embodiment, a method includes establishing a video call between a plurality of client systems, wherein access to an assistant system is persistently maintained during the video call, receiving, from a first client system of the plurality of client systems, a request by a first user to be performed by the assistant system during the video call, wherein the request references one or more activities associated with one or more users associated with the plurality of client systems, analyzing, by a context engine of the assistant system, images of a scene of the video call to identify the one or more activities within the scene, instructing the assistant system to execute the request based on the identified one or more activities, and sending, to one or more of the plurality of client systems, a response to the request while maintaining the video call between the plurality of client systems.

In-call experience enhancement for assistant systems

In one embodiment, a method includes receiving a first user input from a first user, wherein the first user input comprises a partial request, presenting one or more suggested intent auto-completions corresponding to the partial request, receiving a selection by the first user of a first suggested intent auto-completion of the suggested intent auto-completions and a second user input, presenting one or more suggested slot auto-completions corresponding to one or more candidate slot-hypotheses corresponding to the second user input, respectively, wherein each of the candidate slot-hypotheses comprise a slot-suggestion, and wherein each suggested slot auto-completion comprises the second user input and the corresponding candidate slot-hypothesis, receiving a selection by the first user of a first suggested slot auto-completion of the suggested slot auto-completions, and presenting execution results of one or more tasks corresponding to the first suggested intent auto-completion and the first suggested slot auto-completion.

Contextual auto-completion for assistant systems

In one embodiment, a method includes by a client system associated with a user, receiving, at the client system, a user input from the user, parsing, by the client system, the first user input to identify a request to execute a function to be performed by an assistant system of several assistant systems associated with the client system, determining whether the user is authorized to access the assistant system by comparing a voiceprint of the user to several voiceprints stored on the client system, sending, from the client system to the assistant system in response to determining the user is authorized to access the assistant system, a request to set an assistant xbot of the assistant system into a listening mode, and receiving, at the client system from the assistant system, an indication that the assistant xbot is in listening mode.

Multiple wake words for systems with multiple smart assistants

In one embodiment, a method includes receiving, at a client system, an audio input, where the audio input comprises a coreference to a target object, accessing visual data from one or more camera associated with the client system, where the visual data comprises images portraying one or more objects, resolving the coreference to the target object from among the one or more objects, resoling the target object to a specific entity, and providing, at the client system, a response to the audio input, where the response comprises information about the specific entity.

Multimodal entity and coreference resolution for assistant systems

In one embodiment, a method includes receiving a first user request at a client system to suspend a first task being executed by an assistant system operating on the client system, suspending the execution of the first task responsive to the first user request, receiving a second user request at the client system, determining that the second user request is a request to resume the suspended first task based on user interactions with the assistant system with respect to one or more entities associated with the first task, and presenting a prompt to resume the first task at the client system.

User controlled task execution with task persistence for assistant systems

In one embodiment, a method includes, by a client system, receiving, by an assistant xbot of the client system, a request from a first user for a summary of user content from a first content source, retrieving, from the first content source, a plurality of content items corresponding to the request, generating a personalized summary of the retrieved content items, wherein the personalization of the summary is based on a user profile of the first user, and presenting, by the assistant xbot, the personalized summary responsive to the request within a separate communication interface between the assistant xbot and the first user, wherein the personalized summary is interactable by the first user to react to one or more of the plurality of content items.

Content summarization for assistant systems

In one embodiment, a method includes receiving a user query associated with dialog-intents at a client system, executing tasks corresponding to the dialog-intents, generating a multi-perspective response by a stitching model based on two or more of execution results of the tasks, wherein the multi-perspective response comprises a natural-language response combining the two or more execution results, and presenting the multi-perspective response at the client system.

In one embodiment, a method includes receiving, from a client system associated with a user, a user request comprising a reference to a target object, accessing visual data from the client system, wherein the visual data comprises images portraying the target object and one or more additional objects, and wherein attribute information of the target object is recorded in a multimodal dialog state, resolving the reference to the target object based on the attribute information recorded in the multimodal dialog state, determining relational information between the target object and one or more of the additional objects portrayed in the visual data, and sending, to the client system, instructions for presenting a response to the user request, wherein the response comprises the attribute information and the determined relational information.

Multimodal dialog state tracking and action prediction for assistant systems

In one embodiment, a method includes receiving a first user request to execute a first task comprising one or more sub-tasks from a client system associated with a user, receiving a second user request to suspend the first task from the client system, storing a dialog state for the first task in response to the second user request, wherein the dialog state comprises results of executed sub-tasks and a status indicating the first task is suspended, receiving a third user request to resume the first task, accessing the dialog state for the first task, and providing a prompt to resume the first task at the client system.

In one embodiment, a method includes receiving a user request from a client system associated with a user, generating a response to the user request which references one or more entities, generating a personalized recommendation based on the user request and the response, wherein the personalized recommendation references one or more of the entities of the response, and sending instructions for presenting the response and the personalized recommendation to the client system.

Personalized conversational recommendations by assistant systems

In one embodiment, a method includes receiving a query from a user from a client system associated with the user, determining one or more initial memory slots based on the query, accessing a memory graph associated with the user which comprises a plurality of nodes and a plurality of edges connecting the nodes, and wherein one or more of the nodes correspond to one or more episodic memories of the user, respectively, and wherein each edge corresponds to a relationship between the connected nodes, selecting one or more candidate nodes from the memory graph by one or more machine-learning models based on the initial memory slots, generating a response based on the initial memory slots and episodic memories corresponding to the selected candidate nodes, and sending instructions for presenting the response to the client system in response to the query.

Memory grounded conversational reasoning and question answering for assistant systems

Growing community of inventors

San Carlos, CA, United States of America

Rajen Subba