Meta Platforms Technologies, LLC

Facebook, Inc.

Meta Platforms, Inc.

Microsoft Technology Licensing, LLC

Amazon Technologies, Inc.

Facebook Technologies, LLC

Xiaohu Liu

Rajen Subba

Baiyang Liu

Benoit F Dumoulin

Paul Anthony Crook

Ruhi Sarikaya

Francislav Petrov Penov

Zhiguang Wang

Chenlei Guo

Carlos Garcia Jurado Suarez

Fuchun Peng

Mohsen M Agsen

Feras Karablieh

Michael Robert Hanson

Vanessa Y Sun

Ajay M Desai

Kai Jun Weng

Jiedan Zhu

Dilek Zeynep Hakkani-Tur

Gokhan Tur

Larry Paul Heck

Kai Chieh Liu

Xing Fan

Hung The Pham

Joe Pemberton

Vivek Natarajan

Seungwhan Moon

Honglei Liu

Ahmed Aly

Lisa Xiaoyi Huang

Eric Xiao

Chengyuan Ma

Shuting Tang

Shivani Poddar

Sixing Lu

Bigyan Rajbhandari

Krishna Mittal

Zhenyu Yao

Hung Tuan Pham

Zheng Du

Sriharsha Venkata Chintalapati

Shubham Khandelwal

Eun Joon Cho

Xin Ming Fan

Zhongkai Sun

Ali Mamdouh Elkahky

Techniques for determining alternative input representations using entity expansion and entity weighting are described. An entity expansion knowledge base is built by extracting entities from user input-system response pairs that resulted in satisfactory experiences. An extracted entity is associated with an initial score based on it being included in the user input only, in the system response only or both the user input and the system response. Entities co-occurring in the user input-system response pair are connected in the knowledge base. An overall score is associated with the connections based on the initial scores of the connected entities. Using the knowledge base, expansion entities related to an entity included in a user input are determined, and the expansion entities and user input entity are weighted. The weighting of the entities involves assigning a level to each entity based on pairs of user input-alternative input representation.

Alternative input representations

In one embodiment, a method includes receiving a user request from a first user from a client system associated with a first user, wherein the user request comprise a gesture-input from the first user and a speech-input from the first user, determining an intent corresponding to the user request based on the gesture-input by a personalized gesture-classification model associated with the first user, executing one or more tasks based on the determined intent and the speech-input, and sending instructions for presenting execution results of the one or more tasks to the client system responsive the user request.

Personalized gesture recognition for user interaction with assistant systems

In one embodiment, a method includes maintaining contextual information from a first dialog session, wherein the contextual information comprises identifiers of entities and slots, receiving a user request during the first dialog session, determining a context carryover is required for an agent to execute a task associated with the user request, determining a first subset of the identifiers of the entities and a second subset of the slots of the contextual information from the first dialog session are shareable with the agent based on the agent and first resources of the identifiers of the entities and second resources of the slots, and executing the task by the agent based on the first subset of the identifiers of the entities and the second subset of the slots of the shareable contextual information.

Context carryover across tasks for assistant systems

In one embodiment, a method includes establishing a video call between a plurality of client systems, wherein access to an assistant system is persistently maintained during the video call, receiving, from a first client system of the plurality of client systems, a request by a first user to be performed by the assistant system during the video call, wherein the request references one or more activities associated with one or more users associated with the plurality of client systems, analyzing, by a context engine of the assistant system, images of a scene of the video call to identify the one or more activities within the scene, instructing the assistant system to execute the request based on the identified one or more activities, and sending, to one or more of the plurality of client systems, a response to the request while maintaining the video call between the plurality of client systems.

In-call experience enhancement for assistant systems

In one embodiment, a method includes receiving a first user input from a first user, wherein the first user input comprises a partial request, presenting one or more suggested intent auto-completions corresponding to the partial request, receiving a selection by the first user of a first suggested intent auto-completion of the suggested intent auto-completions and a second user input, presenting one or more suggested slot auto-completions corresponding to one or more candidate slot-hypotheses corresponding to the second user input, respectively, wherein each of the candidate slot-hypotheses comprise a slot-suggestion, and wherein each suggested slot auto-completion comprises the second user input and the corresponding candidate slot-hypothesis, receiving a selection by the first user of a first suggested slot auto-completion of the suggested slot auto-completions, and presenting execution results of one or more tasks corresponding to the first suggested intent auto-completion and the first suggested slot auto-completion.

Contextual auto-completion for assistant systems

In one embodiment, a method includes by a client system associated with a user, receiving, at the client system, a user input from the user, parsing, by the client system, the first user input to identify a request to execute a function to be performed by an assistant system of several assistant systems associated with the client system, determining whether the user is authorized to access the assistant system by comparing a voiceprint of the user to several voiceprints stored on the client system, sending, from the client system to the assistant system in response to determining the user is authorized to access the assistant system, a request to set an assistant xbot of the assistant system into a listening mode, and receiving, at the client system from the assistant system, an indication that the assistant xbot is in listening mode.

Multiple wake words for systems with multiple smart assistants

In one embodiment, a method includes maintaining contextual information from a first user request associated with a first task by a context engine, wherein the first task is associated with a first agent, receiving a second user request associated with a second task from a client system, wherein the second user request comprises an ambiguous mention and the second task is associated with a second agent, determining a context carryover is required for the second agent to execute the second task, determining the ambiguous mention corresponds to one or more data items associated with the contextual information from the first user request, and providing the one or more data items to the second agent for execution of the second task.

In one embodiment, a method includes receiving a user input including an ambiguous mention by a user from the user's client system, analyzing the user input to identify intents and slots based on a natural-language understanding module, where at least one slot is an ambiguous slot corresponding to the ambiguous mention, identifying candidate entities corresponding to the ambiguous slot, ranking the candidate entities based on the user's user profile data, sending to the user's client system instructions for prompting the user to select a candidate entity from a set of candidate entities having a rank greater than a threshold rank, where the selected candidate entity is resolved to the ambiguous slot, and storing the selected candidate entity together with the ambiguous mention as a mention-entity pair as part of the user profile data responsive to receiving the selection from the user.

Disambiguating user input with memorization for improved user assistance

In one embodiment, a method includes receiving a user request from a client system associated with a user, determining that the user request corresponds to a first suspended task, retrieving a first dialog state of the first suspended task from a dialog history associated with the user, generating a summary of the first suspended task based on the first dialog state using a natural-language generating (NLG) module, and sending instructions to the client system for providing the summary of the first suspended task to the user.

Conversation summarization during user-control task execution for assistant systems

Techniques for performing centralized unsuperivised learning in a multi-domain system are described. A user may request labeled data for an ML task, where the request includes a prompt for obtaining relevant explicit user feedback. The system may use the prompt to collect explicit user feedback for relevant runtime user inputs. After a duration of time (in the user's request for labeled data) has elapsed, the system determines whether collected user feedback indicates processing of the user input was defective and, if so, determines a cause of the defective processing. The system then uses one or more label generators to generate labeled data using the collected user feedback, whether the processing was defective, and the determined defect cause.

Machine learning label generation

In one embodiment, a method includes receiving a user request from a first user at a client system, wherein the user request is associated with a semantic-intent, identifying dialog-intents associated with the user request by the client system based on the semantic-intent and context information associated with the user request, wherein each dialog-intent is a sub-intent of the semantic-intent; determining agents for executing tasks associated with the dialog-intents by the client system, and presenting information returned from the agents responsive to executing the tasks at the client system.

Intent identification for agent matching by assistant systems

In one embodiment, a method includes receiving a first user request at a client system to suspend a first task being executed by an assistant system operating on the client system, suspending the execution of the first task responsive to the first user request, receiving a second user request at the client system, determining that the second user request is a request to resume the suspended first task based on user interactions with the assistant system with respect to one or more entities associated with the first task, and presenting a prompt to resume the first task at the client system.

User controlled task execution with task persistence for assistant systems

In one embodiment, a method includes, by a client system, receiving, by an assistant xbot of the client system, a request from a first user for a summary of user content from a first content source, retrieving, from the first content source, a plurality of content items corresponding to the request, generating a personalized summary of the retrieved content items, wherein the personalization of the summary is based on a user profile of the first user, and presenting, by the assistant xbot, the personalized summary responsive to the request within a separate communication interface between the assistant xbot and the first user, wherein the personalized summary is interactable by the first user to react to one or more of the plurality of content items.

Content summarization for assistant systems

Growing community of inventors

Bellevue, WA, United States of America

Xiaohu Liu