Facebook, Inc.

Facebook Technologies, LLC

Meta Platforms, Inc.

Hermes Germi Pique Corchs

Razvan Gabriel Racasanu

Colin Todd Miller

Kirill A Pugin

Trevor Charles Armstrong

Pradeep George Mathias

Mauricio Narvaez

Gwylim Aidan Ashley

Mykyta Lutsenko

Tomer Bar

Ragavan Srinivasan

Miguel Goncalves

Bryce David Redd

Marcin Kwiatkowski

Mark I-Kai Wang

Ficus Kirkpatrick

William Samuel Bailey

Balamanohar Paluri

Alvaro Collet Romea

Krishnan Ramnath

Ryan Keenan Olson

Lada Ariana Adamic

Matthew Roberts

Steven Kish

Jeffrey C Wieland

Brett Alden Lavalla

Tullie Murrell

Shaomei Wu

Olivier Marie Bouan Du Chef Du Bos

Matt King

Houman Meshkin

Thomas Ward Meyer

Omid Farivar

Jim Sing Liu

Jiao Li

Alvarao Collet Romea

In one embodiment, the system may receive a serialized data stream generated by serializing data chunks including data from a video stream and contextual data streams associated with the video stream. The contextual data streams may include a first computed data stream and a sensor data stream. The system may extract the video data stream and one or more contextual data streams from the serialized data stream. The system may generate a second computed data stream based on the sensor data stream in the extracted contextual data streams. The system may compare the second computed data stream to the first computed data stream extracted from the serialized data stream to select a computed data stream based on one or more pre-determined criteria. The system may render an artificial reality effect for display with the extracted video data stream based at least in part on the selected computed data stream.

Data sterilization for post-capture editing of artificial reality effects

In one embodiment, a method includes a system accessing an image, which may comprise covered and uncovered portions, and an overlay image comprising opaque pixels. The covered portion may be configured to be covered by the opaque pixels of the overlay image. The system may generate a data structure comprising data elements associated with pixels of the image. Each of the data elements associated with a covered pixel in the covered portion of the image may be configured to identify an uncovered pixel in the uncovered portion of the image that is closest to the covered pixel. Each covered pixel in the covered portion of the image may be modified by accessing the data element associated with the covered pixel, determining a distance between the covered pixel and an associated closest uncovered pixel using the accessed data element, and modifying a color of the covered pixel based on the distance.

Systems and methods for occluding images and videos subject to augmented-reality effects

The disclosed computer-implemented method may include receiving a first input from a first artificial reality device detecting a first environment of a first user and determining a first environmental feature of the first environment based on the first input. The method may include receiving a second input from a second artificial reality device detecting a second environment of a second user and determining a second environmental feature of the second environment based on the second input. The method may include comparing the first environmental feature with the second environmental feature and including, based on the comparison, the first and second users in a group for online interactions. Various other methods, systems, and computer-readable media are also disclosed.

User groups based on artificial reality

In one embodiment, the system captures a video data stream of a scene and contextual data streams associated with the video data stream. The contextual data streams comprise a sensor data stream or a computed data stream. The system renders an artificial reality effect based on the contextual data streams for display with the video data stream. The system generates a serialized data stream by serializing data chunks of the video data stream and the contextual data streams. The system stores the serialized data stream into a storage. The system extracts the video data stream and one or more of the contextual data streams from the serialized data stream by deserializing the data chunks in the serialized data stream. The system renders the same or another artificial reality effect for display with the extracted video data stream based on the extracted contextual data streams.

A video effects application executes on a client device having an image capture device and receives video data captured by the image capture device. The video effects application extracts information from the captured video data and stores the extracted information as metadata associated with the captured video data. For example, the video effects application identifies objects in the captured video data or identifies optical flow of the captured video data and stores the identified objects or identified optical flow as metadata associated with the captured video data. The video effects application stores information describing modifications to the captured video data in association with the captured video data. When the captured video data is presented, the captured video data, associated metadata, and information describing the modifications is communicated to a renderer, which uses the metadata to perform the identified modifications to the captured video data when presenting the captured video data.

Architecture for augmenting video data obtained by a client device with one or more effects during rendering

In one embodiment, a method includes generating, by a device, first tracking data using a first tracking algorithm, based on first video frames associated with a scene. An augmented-reality (AR) effect may be displayed based on the first tracking data. The device may generate a first confidence score associated with the first tracking data and determine that the first confidence score is above a threshold. The device may generate, based on second video frames subsequent to the first video frames, second tracking data using the first tracking algorithm. The device may determine that an associated second confidence score is below a threshold. In response, the device may generate, based on third video frames subsequent to the second video frames, third tracking data using a second tracking algorithm different from the first. The device may then display the AR effect based on the third tracking data.

Dynamic graceful degradation of augmented-reality effects

In one embodiment, a computing system captures, using a camera, a number of frames of a live scene. The system generates preview frames for an augmented scene by applying one or more augmented reality effects to the frames of the live scene. Each preview frame is based on a low-resolution image of the live scene. The low-resolution image has a lower resolution than a maximum resolution of the camera. The system stores at least one preview frame with the augmented reality effects into a storage of the computing device. The system displays a live preview of the augmented scene using the preview frames. The system receives a request from a user to capture an image of the augmented scene while the live preview is being displayed. The system retrieves, the at least one preview frame stored in the storage and outputs the retrieved at least one preview frame.

Systems and methods for capturing images with augmented-reality effects

A media effects engine on a computer device applies one or more effects to an input media stream. A performance monitor monitors a performance metric associated with playing the input media stream and reduces a quality parameter associated with the effect upon detecting a drop in the performance metric below a target metric. The quality parameter manages a tradeoff between a quality of effect and an amount of hardware resources consumed to produce the effect. Thus, the effect can be adjusted to meet the capabilities of the computer device.

Dynamic control of media effects based on hardware performance

In one embodiment, a method for presenting an augmented reality effect may include receiving, by a computing system, a request for downloading an augmented reality effect, which may include a plurality of elements. The system may select a first subset of elements among the plurality of elements based on one or more predefined rules. The first subset of elements may be transmitted to the client device for display. The system may transmit a remaining subset of elements of the plurality of elements to the client device for display after the transmitting of the first subset of elements is complete. The augmented reality effect may be configured to launch and display at least one element of the first subset of elements prior to the remaining subset of elements is received by the client device.

Systems and methods for incrementally downloading augmented-reality effects

Systems, methods, and non-transitory computer-readable media can receive an image. One or more concepts depicted in the image are identified based on machine learning techniques. The one or more concepts are filtered based on filtering criteria to identify one or more selected concepts. An image description is generated comprising the one or more selected concepts.

Systems and methods for image description generation

In one embodiment, a method for distributing an augmented reality (AR) effect may include ranking, by a computing system, a plurality of client devices based on a device capability, where a higher ranking may represent relatively better device capability. The system may identify a first subset of client devices having a first particular ranking. An AR effect may be transmitted to the first subset of client devices. The system may determine a performance score for the AR effect among the first subset of client devices. The performance score may be based on one or more performance parameters associated with the AR effect running on the first subset of client devices. The system may transmit the AR effect to a second subset of client devices when the performance score of the AR effect among the first subset of client devices satisfies a predefined threshold.

Systems and methods for distributing augmented-reality effects

Growing community of inventors

Menlo Park, CA, United States of America

Hermes Germi Pique Corchs