Facebook Technologies, LLC

Meta Platforms Technologies, LLC

Lei Xiao

Douglas Robert Lanman

Alexander Jobe Fix

Anton S Kaplanyan

Matthew Steven Chapman

Andrew Maimone

Marina Zannoli

Joel Hegland

Kara Emery

Robin Joseph Swanson

Salah Eddine Nouri

According to examples, a head-mounted display (HMD) headset may include at least two exterior-facing color cameras mounted on the front face of the HMD and in the same visual plane as the user's eyes. The at least two exterior-facing color cameras may collect images for stereo view synthesis. The HMD may include a processor, and a memory storing instructions, which when executed by the processor, cause the processor to perform stereo view synthesis with a machine-learning (ML) based technique including at least one of depth estimation, imaging sharpening, forward splatting, disocclusion filtering, or fusion.

Display system with machine learning (ML) based stereoscopic view synthesis over a wide field of view

In one embodiment, a method includes receiving a first frame associated with a first time and one or more second frames of a video having a resolution lower than a target resolution, wherein each second frame is associated with a second time prior to the first time, generating a first feature map for the first frame and one or more second feature maps for the one or more second frames, up-sampling the first feature map and the one or more second feature maps to the target resolution, warping each of the up-sampled second feature maps according to a motion estimation between the associated second time and the first time, and generating a reconstructed frame having the target resolution corresponding to the first frame by using a machine-learning model to process the up-sampled first feature map and the one or more up-sampled and warped second feature maps.

Neural super-sampling for real-time rendering

In one embodiment, a computing system may generate and display a virtual reality environment to a user. The computing system may determine a head pose of the user based on headset tracking data associated with a headset worn by the user. The computing system may determine a hand pose of the user based on hand tracking data associated with a device held or worn by a hand of the user. The computing system may access scene information associated with the displayed virtual reality environment. The computing system may determine a predicted focal point of the user within the virtual reality environment by processing the head pose, the hand pose, and the scene information using a machine-learning model.

Using deep learning to determine gaze

In one embodiment, a system may generate a hologram by processing a first image using a machine-learning model. The system may generate a second image based on at least a portion of the hologram using a processing model that is configured to simulate interactions between a light source and the hologram. The system may compare the second image to the first image to calculate a loss based on a loss function. The system may update the machine-learning model based on the loss between the first image and the second image. The updated machine-learning model is configured to process one or more input images to generate one or more corresponding holograms.

Systems and methods for generating holograms using deep learning

In one embodiment, a computing system may receive current eye-tracking data associated with a user of a head-mounted display. The system may dynamically adjust a focal length of the head-mounted display based on the current eye-tracking data. The system may generate an in-focus image of a scene and a corresponding depth map of the scene. The system may generate a circle-of-confusion map for the scene based on the depth map. The circle-of-confusion map encodes a desired focal surface in the scene. The system may generate, using a machine-learning model, an output image with a synthesized defocus-blur effect by processing the in-focus image, the corresponding depth map, and the circle-of-confusion map of the scene. The system may display the output image with the synthesized defocus-blur effect to the user via the head-mounted display having the adjusted focal length.

Systems and methods for generating defocus blur effects

In one embodiment, a system may access a training sample that includes training images and corresponding training depth maps of a scene, with the training images being associated with different predetermined viewpoints of the scene. The system may generate elemental images of the scene by processing the training images and the training depth maps using a machine-learning model. The elemental images are associated with more viewpoints of the scene than the predetermined viewpoints associated with the training images. The system may update the machine-learning model based on a comparison between the generated elemental images of the scene and target elemental images that are each associated with a predetermined viewpoint. The updated machine-learning model is configured to generate elemental images of a scene of interest based on input images and corresponding depth maps of the scene of interest from different viewpoints.

Systems and methods utilizing a machine learning model for generating defocus blur effects

In one embodiment, a system may access a training sample from a training dataset, including a training image of a scene and a corresponding depth map. The system may access a circle-of-confusion map for the scene, which is generated based on the depth map and encodes a desired focal surface in the scene. The system may generate an output image by processing the training image, the corresponding depth map, and the corresponding circle-of-confusion map using a machine-learning model. The system may update the machine-learning model based on a comparison between the generated output image and a target image depicting the scene with a desired defocus-blur effect. The updated machine-learning model is configured to generate images with defocus-blur effect based on input images and corresponding depth maps.

In one embodiment, a system may access a training sample from a training dataset. The training sample may include a training image of a scene and a corresponding depth map of the scene. The system may generate a plurality of decomposition images by processing the training image and the corresponding depth map using a machine-learning model. The system may generate a focal stack based on the plurality of decomposition images and update the machine-learning model based on a comparison between the generated focal stack and a target focal stack associated with the training sample. The updated machine-learning model is configured to generate decomposition images with defocus-blur effect based on input images and corresponding depth maps.

Growing community of inventors

Redmond, WA, United States of America

Lei Xiao