Meta Platforms Technologies, LLC

Facebook Technologies, LLC

Facebook, Inc.

Meta Platforms, Inc.

James Allan Booth

Gioacchino Noris

Alexander Sorkine Hornung

Alessia Marra

Panya Inversin

Alexandru-Eugen Ichim

Gaurav Chaurasia

Manuel Werlberger

Sarthak Ray

Elif Albuz

Peihong Guo

Tong Xiao

Alex Locher

Mahdi Salmani Rahimi

The disclosed artificial reality system can provide a user self representation in an artificial reality environment based on a self portion from an image of the user. The artificial reality system can generate the self representation by applying a machine learning model to classify the self portion of the image. The machine learning model can be trained to identify self portions in images based on a set of training images, with portions tagged as either depicting a user from a self-perspective or not. The artificial reality system can display the self portion as a self representation in the artificial reality environment by positioning them in the artificial reality environment relative to the user's perspective in the artificial reality environment. The artificial reality system can also identify movements of the user and can adjust the self representation to match the user's movement, providing more accurate self representations.

Self presence in artificial reality

In one embodiment, one or more computing systems may access a plurality of images corresponding to a portion of a face of a user. The plurality of images is captured from different viewpoints by a plurality of cameras coupled to an artificial-reality system worn by the user. The one or more computing systems may synthesize, using a machine-learning model, to generate a synthesized image corresponding to the portion of the face of the user. The one or more computing systems may access a three-dimensional (3D) facial model representative of the face of the user and generate a texture image by projecting the synthesized image onto the 3D facial model from a specific camera pose. The one or more computing systems may cause an output image of a facial representation of the user to be rendered using at least the 3D facial model and the texture image.

Camera reprojection for faces

A mixed reality (MR) simulation system includes a console and a head mounted device (HMD). The MR system captures stereoscopic images from a real-world environment using outward-facing stereoscopic cameras mounted to the HMD. The MR system preprocesses the stereoscopic images to maximize contrast and then extracts a set of features from those images, including edges or corners, among others. For each feature, the MR system generates one or more two-dimensional (2D) polylines. Then, the MR system triangulates between 2D polylines found in right side images and corresponding 2D polylines found in left side images to generate a set of 3D polylines. The MR system interpolates between 3D vertices included in the 3D polylines or extrapolates additional 3D vertices, thereby generating a geometric reconstruction of the real-world environment. The MR system may map textures derived from the real-world environment onto the geometric representation faster than the geometric reconstruction is updated.

Reconstruction of essential visual cues in mixed reality applications

In one embodiment, one or more computing systems may receive an image of a portion of a face of a first user. The one or more computing systems may access a three-dimensional (3D) facial model representative of the face of the first user. The one or more computing systems may identify one or more facial features captured in the image and determine a camera pose relative to the 3D facial model based on the identified one or more facial features in the image and predetermined feature locations on the 3D facial model. The one or more computing systems may determine a mapping relationship between the image and the 3D facial model by projecting the image of the portion of the face of the first user onto the 3D facial model from the camera pose and cause an output image of a facial representation of the first user to be rendered.

In one embodiment, a method includes receiving image data corresponding to an external environment of a user. The image data is captured at a first time and comprises a body part of the user. The method also includes receiving a first tracking data generated based on measurements made at the first time by at least one motion sensor associated with the body part; generating, based at least on the image data, a model representation associated with the body part; receiving a second tracking data generated based on measurements made at a second time by the at least one motion sensor associated with the body part; determining a deformation of the model representation associated with the body part based on the first tracking data and the second tracking data; and displaying the deformation of the model representation associated with the body part of the user.

Passthrough visualization

In one embodiment, a method includes displaying a first virtual content to a first user in a virtual area, the virtual area comprising one or more second virtual content, inferring an intent of the first user to interact with the first virtual content based on one or more of first user actions or contextual information, and adjusting one or more configurations associated with one or more of the second virtual content based on the inferring of the intent of the first user to interact with the first virtual content.

Virtual spaces, mixed reality spaces, and combined mixed reality spaces for improved interaction and collaboration

In one embodiment, a method includes receiving, from the first user, a request to create a joint virtual space to use with one or more second users, determining a first area in a first room associated with the first user based at least in part on space limitations associated with the first room and locations of one or more items in the first room, retrieving information associated with one or more second rooms for each of the second users, creating, based on first area of the first room and the information associated with each of the second rooms, the joint virtual space, and providing access to the joint virtual space to the first user and each of the one or more second users.

Growing community of inventors

Zurich, Switzerland

James Allan Booth