Facebook, Inc.

Meta Platforms, Inc.

Meta Platforms Technologies, LLC

Facebook Technologies, LLC

Vincent Charles Cheung

Connie Yeewei Ho

Daniel Tsuan

Hyunbin Park

Maheen Sohail

Chengxuan Bai

Balmanohar Paluri

Rafael Camargo

Girish Patangay

Tali Zvi

Olivier Charles Gratry

Jiemin Zhang

Ruoni Wang

Nipun Mathur

Viresh Rustagi

James Glessal Pearce

Yating Sheng

Sergiu Dogaru

Kent Austin White

Andrew Pitcher Thompson

Marc Layne Hemeon

Olivier Sebastien Joseph Bartholot

Baback Elmieh

Stef Marc Smet

Bradley Duane Kowalk

Mark Alexander Walsh

Hermes Germi Pique Corchs

Maria Chiara Cacciani

Andrew James Thomas Buckley

Meng Wang

Milen Georgiev Ozhumerov

Mircea-Gabriel Suciu

Muhammed Elsayed Muhammed Elsayed Ibrahim

Cunpu Bo

Kevin Lin

A computer implemented method for location-based interaction in an artificial reality (XR) environment comprises determining an indication of a location. A historical state of the location is retrieved in the XR environment. One or more inputs specifying a command related to the historical state of the location, are received on an XR device. The command indicates either to retrieve information of virtual entities that previously existed at the location or to undo a modification of one or more features at the location. One or more virtual entities are matched to the location in the XR environment for a given timeframe and an indication of the matched one or more virtual entities, is displayed.

Artificial reality (XR) location-based displays and interactions

Aspects of the present disclosure are directed to an artificial intelligence ('AI') application running in conjunction with an artificial reality (“XR”) space. The AI Builder responds to user commands, verbal or gestural, to build or edit spaces or objects in space. If the requested object is of a type recognized by the AI Builder, then the AI Builder builds the object from one or more stored templates. The new object's location is determined by the objects that already exist in the user's XR environment and on commands or gestures from the user. If the AI Builder does not recognize the requested object, the user can show an image to the AI Builder, and the AI builds a 3D object in the XR space according to that image. To ease collaboration among users, the AI Builder may present its user interface as a non-player character within the XR world.

Artificial intelligence-assisted virtual object builder

The present embodiments relate to automated memory creation and retrieval from moment content items. In some implementations, the automated memory creation and retrieval system can obtain moment content items from user-designated sources with a single user perspective or multiple user perspectives. The moment content items can be assigned tags and arranged in chronological order. The arranged moment content items can be clustered into memory content items based on clustering conditions. Once memory content items are created, they can be arranged into a memory hierarchy made up of connected nodes. In some implementations, the memory content items are also linked together based on similarity in various dimensions in a memory graph. The automated memory creation and retrieval system can receive search criteria for memories from a user interface and provide the user with memories from matched nodes in the memory hierarchy or linked memories in the memory graph.

Automated memory creation and retrieval from moment content items

A conversation augmentation system can automatically augment a conversation with content items based on natural language from the conversation. The conversation augmentation system can select content items to add to the conversation based on determined user 'intents' generated using machine learning models. The conversation augmentation system can generate intents for natural language from various sources, such as video chats, audio conversations, textual conversations, virtual reality environments, etc. The conversation augmentation system can identify constraints for mapping the intents to content items or context signals for selecting appropriate content items. In various implementations, the conversation augmentation system can add selected content items to a storyline the conversation describes or can augment a platform in which an unstructured conversation is occurring.

Automated conversation content items from natural language

A multimedia query system is described that includes a multimedia capture system configured to capture raw multimedia data comprising at least one of raw video data or raw audio data, a metadata engine configured to extract one or more anchor points of metadata from the raw multimedia data and to store the one or more anchor points of metadata, wherein the anchor points of metadata includes references to respective portions of the raw multimedia data. The multimedia query system further includes a storage engine configured to store the raw multimedia data, a recall engine configured to receive a query and to apply the query to the one or more anchor points of metadata to identify one or more raw multimedia data candidates from the portions of the raw multimedia data, and a query engine configured to generate a response to the query based on the one or more raw multimedia data candidates.

Multimedia query system

This disclosure describes transcribing speech using audio, image, and other data. A system is described that includes an audio capture system configured to capture audio data associated with a plurality of speakers, an image capture system configured to capture images of one or more of the plurality of speakers, and a speech processing engine. The speech processing engine may be configured to recognize a plurality of speech segments in the audio data, identify, for each speech segment of the plurality of speech segments and based on the images, a speaker associated with the speech segment, transcribe each of the plurality of speech segments to produce a transcription of the plurality of speech segments including, for each speech segment in the plurality of speech segments, an indication of the speaker associated with the speech segment, and analyze the transcription to produce additional data derived from the transcription.

Speech transcription using multiple data sources

This disclosure describes a computing system that automatically detects users in visual proximity and adds the users to a private collaboration space enabling the users to share digital content. In one example, the computing system includes a video processing engine configured to detect, from first image data representative of a first physical environment that includes a second user, the second user, wherein the first image data is captured by an image capture system of a head-mounted display (HMD) worn by a first user. The computing system also includes a collaboration application configured to add, in response to detection of the second user, the second user to a set of users associated with a private collaboration space in which the set of users access shared digital content, wherein the set of users includes the first user.

Private collaboration spaces for computing systems

Techniques for enhancing group communication on a device are described. A method may include presenting a message in a message portion of a user interface (UI) for a group communication application executing on a first device, where the group communication application communicates messages among multiple devices. The method may further include monitoring an input component for the first device and detecting a first control directive from the input component. The first control directive may select a message displayed in the message portion of the user interface. The method may further include performing a pin operation on the selected message; and presenting the pinned message in a banner portion of the UL Other embodiments are described and claimed.

Techniques for enhancing group communication on a mobile device

Various client devices include displays and one or more image capture devices configured to capture video data. Different users of an online system may authorize client devices to exchange information captured by their respective image capture devices. Additionally, a client device modifies captured video data based on users identified in the video data. For example, the client device changes parameters of the image capture device to more prominently display a user identified in the video data and may further change parameters of the image capture device based on gestures or movement of the user identified in the video data. The client device may apply multiple models to captured video data to modify the captured video data or subsequent capturing of video data by the image capture device.

Modifying capture of video data by an image capture device based on video data previously captured by the image capture device

In one embodiment, a method includes detecting objects in an image. The method includes accessing a mask for each object. The method includes receiving an input in relation to the image. The input corresponds to an input region and an input type. The method includes identifying a region of the image corresponding to the input region of the input. The identified region of the image includes one or more of the masks. The method includes providing feedback regarding the one or objects in the identified region of the image based on the input type.

Image segmentation with touch interaction

Various client devices include displays and one or more image capture devices configured to capture video data. Different users of an online system are associated with client devices that exchange information captured by their respective image capture devices. A receiving client device includes information identifying one or more users of an online system authorized to exchange video data with the receiving client device when the receiving client device determines a receiving user is within a field of view of the receiving client device. Video data is exchanged between the receiving client device and a client device associated with a user authorized to exchange video data without the receiving user providing an input to the receiving client device to accept the video data exchange.

Exchange content between client devices when a client device determines a user is within a field of view of an image capture device of the client device and authorized to exchange content

Various client devices include displays and one or more image capture devices configured to capture video data. Different users of an online system are associated with client devices that exchange information captured by their respective image capture devices. When exchanging information, presentation of content to users associated with different client device may be initially synchronized across the client devices. To synchronize content presentation, a client device initiating presentation of the content transmits a request identifying the content and an initial time to other client devices. The initial time is greater than a maximum return time or latency in a network coupling the client devices and the online system from a time when the request is transmitted. A client device determined to be out of synchronization with one or more other client devices receives a command to modify a rate at which the content is presented to reestablish synchronization.

Synchronizing presentation of content presented by multiple client devices

Growing community of inventors

San Carlos, CA, United States of America

Vincent Charles Cheung