Facebook, Inc.

Meta Platforms, Inc.

Google Inc.

Vivek Natarajan

Shawn C P Mei

Baiyang Liu

Zhengping Zuo

Wenhai Yang

Brian Nelson

Scott Martin

Krishna Mittal

Xiaohu Liu

Laurent Nicolas Landowski

Shubham Gupta

Sylvain Gelly

Neil Matthew Tinmouth Houlsby

Julia Anne Framel

Ahmed Aly

Alexander Kolesnikov

Lucas Klaus Beyer

Xiaohua Zhai

Basil Mustafa

Joan Puigcerver I Perez

Yin Ching Jessica Yung

Alan Prasana Karthikesalingam

Jan Freyberg

Wen Yau Aaron Loh

Patricia Leigh MacWilliams

Davide Testuggine

Brett Charles Groel

In one embodiment, a method includes receiving at a head-mounted device a speech input from a user and a visual input captured by cameras of the head-mounted device, wherein the visual input comprises subjects and attributes associated with the subjects, and wherein the speech input comprises a co-reference to one or more of the subjects, resolving entities corresponding to the subjects associated with the co-reference based on the attributes and the co-reference, and presenting a communication content responsive to the speech input and the visual input at the head-mounted device, wherein the communication content comprises information associated with executing results of tasks corresponding to the resolved entities.

Processing multimodal user input for assistant systems

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for training a neural network to perform a downstream computer vision task. One of the methods includes pre-training an initial neural network that shares layers with the neural network to perform an initial computer vision task and then training the neural network on the downstream computer vision task.

Transfer learning between different computer vision tasks

In one embodiment, a method includes, by one or more computing systems, receiving, by an assistant xbot associated with the one or more computing systems, a user request for a content digest, retrieving one or more content objects corresponding to the user request, generating one or more slides for the one or more retrieved content objects, respectively, and providing, by the assistant xbot, instructions for presenting the content digest responsive to the request from the first user, wherein the content digest comprises the one or more slides, and wherein one or more of the slides of the content digest are removed from the content digest after a predetermined time period.

Ephemeral content digests for assistant systems

In one embodiment, a method includes receiving, from a client system of a user, a user input comprising a plurality of n-grams, parsing the user input to identify one or more overall intents, hidden intents, and slots associated with the one or more n-grams, wherein at least one of the hidden intents is non-resolvable for being associated with partial slot information corresponding to an n-gram that has not been resolved to a particular entity identifier, wherein the partial slot information is associated with two more entity identifiers of two or more entities, respectively, sending, to the client system, instructions for prompting the user to select one of the entities to be associated with the non-resolvable hidden intent, resolving the non-resolvable hidden intent based on the entity identifier of the entity selected by the user, and generating a response to the user input based on the resolved hidden intent.

Execution engine for compositional entity resolution for assistant systems

In one embodiment, a method includes receiving a user input based on a plurality of modalities at the client system, wherein at least one of the modalities of the user input is a visual modality, determining one or more subjects and one or more attributes associated with the one or more subjects, respectively, based on the visual modality of the user input, resolving one or more entities corresponding to the one or more subjects based on the determined one or more attributes, and presenting a communication content at the client system responsive to the user input, wherein the communication content comprises information associated with executing results of one or more tasks corresponding to the one or more resolved entities.

In one embodiment, a method includes, by one or more computing systems, receiving a user input comprising a plurality of n-grams from a user of a client system, generating a tree-structured representation for the user input based on a parsing by a compositional model, resolving the tree-structured representation by applying a depth-first search algorithm, wherein the tree-structured representation comprises one or more non-resolvable non-terminal nodes associated with one or more slots, and wherein each non-terminal parent node of a non-resolvable non-terminal node is partially resolved based on partial slot information associated with the non-resolvable non-terminal node, and wherein each non-resolvable non-terminal node is resolved based on the respective partially resolved non-terminal parent node of the non-resolvable non-terminal node, generating a response to the user input based on the resolved tree-structured representation, sending instructions for presenting the response to the client system of the user.

In one embodiment, a method includes receiving a user input by the first user from a client system associated with a first user, parsing the user input to identify one or more n-grams associated with the user input, accessing a user profile associated with the first user, wherein the user profile is stored in a first data store, accessing ontology data based on the one or more identified n-grams from one or more information graphs, wherein the one or more information graphs are stored in one or more second data stores, respectively, determining contextual information associated with the user input, generating semantic information by aggregating the user profile, ontology data, and contextual information, generating a feature representation for the identified one or more n-grams based on the semantic information, and resolving one or more entities associated with the one or more n-grams based on the feature representation.

Aggregating semantic information for improved understanding of users

In one embodiment, a method includes, by one or more computing systems, receiving, from a client system associated with a user, a request from the user for a content digest from an online social network, retrieving one or more content objects associated with the online social network that are accessible by the user, selecting one or more of the retrieved content objects to incorporate into the content digest based on their identified categories, generating one or more slides for the one or more selected content objects, respectively, wherein each slide comprises a summary and representative image of the respective selected content object, sending, to the client system of the user, instructions for presenting the content digest responsive to the request from the user, wherein the content digest comprises the one or more slides.

In one embodiment, a method includes receiving from a client system associated with a first user a user input based on one or more modalities, at least one of which is a visual modality, identifying one or more subjects associated with the user input based on the visual modality based on one or more machine-learning models, determining one or more attributes associated with the one or more subjects respectively based on the one or more machine-learning models, resolving one or more entities corresponding to the one or more subjects based on the determined one or more attributes, executing one or more tasks associated with the one or more resolved entities, and sending instructions for presenting a communication content including information associated with the executed one or more tasks responsive to user input to the client system associated with the first user.

A virtual assistant receives a message including message content from a client device. The virtual assistant determines an intent to organize an event and initial parameters for the event based on the message content. The virtual assistant retrieves a set of messages related to the received message from a data store and refines the initial parameters based on the related messages. A set of potential recommendations is generated based on the refined event parameters and the virtual assistant selects one or more of the potential recommendations to surface to users. The selected recommendations are sent to the client device for presentation to the user.

Growing community of inventors

Palo Alto, CA, United States of America

Vivek Natarajan