Meta Platforms Technologies, LLC

Facebook Technologies, LLC

Olympus Corporation

Leland Stanford Junior University

Steven Paul Lansel

Sebastian Sztuk

Javier San Agustin Lopez

Todd Douglas Keeler

Rohit Rao Padebettu

Behnam Bastani

Brian A Wandell

Brian M Wheelwright

Haomiao Jiang

Jian Zhang

Andy Lai Lin

Kirk Eric Burgess

Guodong Rong

Grant Kaijuin Yang

In one embodiment, a method may obtain, from an application, (a) an image and (b) a layer frame having a first pose in front of the image. The method may generate, for a first viewpoint associated with a first time, a first display frame by separately rendering the image and the layer frame having the first pose into a display buffer. The method may display the first display frame at the first time. The method may determine an extrapolated pose for the layer frame based on the first pose of the layer frame and a second pose of a previously submitted layer frame. The method may generate, for a second viewpoint associated with a second time, a second display frame by separately rendering the image and the layer frame having the extrapolated pose into the display buffer. The method may display the second display frame at the second time.

Compositor layer extrapolation

In one embodiment, a method includes accessing a first image corresponding to a first frame of a video stream, where the first image has complete pixel information, rendering a provisional image corresponding to a second frame of the video stream subsequent to the first frame, where the provisional image has a first area with complete pixel information and a second area with incomplete pixel information, generating a predicted image corresponding to the second frame by re-projecting at least an area of the first image according to one or more warping parameters, and generating a second image corresponding to the second frame by compositing the rendered provisional image and the predicted image.

Temporal foveated rendering

In particular embodiments, the disclosure provides a method comprising: rendering, on a graphics processing unit (GPU), a low-resolution image associated with a scene, the low-resolution image having a resolution that is lower than a target resolution; transmitting a version of the low-resolution image to a neural accelerator; processing, on the neural accelerator, the version of the low-resolution image using a trained machine-learning model, thereby outputting a plurality of control parameters; transmitting the control parameters from the neural accelerator to the GPU; processing, on the GPU, the low-resolution image and the control parameters to construct a high-resolution image having the target resolution, wherein the GPU is programmed to determine a plurality of pixel weights for performing an interpolation using the control parameters; and outputting the high-resolution image.

Split machine-learning assisted graphics rendering pipeline

A method for evaluating an external machine learning program while limiting access to internal training data includes providing labeled training data from a first source, receiving, by the first source, a machine learning program from a second source different from the first source, blocking, by the first source, access by the second source to the labeled training data, and training, by the first source, the machine learning program according to a supervised machine learning process using the labeled training data. The method further includes generating a first set of metrics from the supervised machine learning process that provide feedback about training of the neural network model, analyzing the first set of metrics to identify subset data therein, and, in order to permit evaluation of the neural network model, transmitting, to the second source, those metrics from the first set of metrics that do not include the subset data.

System and method for controlling confidential information

In one embodiment, a method includes receiving a source image and its associated parameters from each of multiple image sources, associating each of the source images with a layer in a range of layers based on the parameters associated with the source images, the range of layers specifying a composition layering order of the source images, generating a corresponding customized distortion mesh for each particular source image in the source images based on the parameters associated with the particular source image and at least a portion of the parameters associated with each of the source images that is associated with any layer preceding a layer associated with the particular source image, modifying each of the source images using the corresponding customized distortion mesh, generate a composite image using the modified source images, and display the composite image as a frame in a video.

Generating a composite image

Various aspects of the subject technology relate to prediction of eye movements of a user of a head-mountable display device. Predictive foveated display systems and methods, using the predicted eye movements are also disclosed. Predictive variable focus display systems and methods using the predicted eye movements are also disclosed. Predicting eye movements may include predicting a future gaze location and/or predicting a future vergence plane for the user's eyes, based on the current motion of one or both of the user's eyes. The predicted gaze location may be used to pre-render a foveated display image frame with a high-resolution region at the predicted gaze location. The predicted vergence plane may be used to modify an image plane of a display assembly to mitigate or avoid a vergence/accommodation conflict for the user.

Predictive eye tracking systems and methods for foveated rendering for electronic displays

The disclosed computer-implemented method may include (1) displaying one or more images to a user via a display comprising multiple display regions, (2) switching each of the display regions to a blocking state in which a view of the user's real-world environment in a corresponding region of the user's field of view is blocked from the user, (3) detecting a pass-through triggering event involving one or more objects in the user's real-world environment, (4) identifying one or more display regions corresponding to a region of the user's field of view occupied by the object, and (5) switching each of the one or more display regions to a pass-through state in which the view of the user's real-world environment in the corresponding region of the user's field of view is passed through to the user. Various other methods, systems, and computer-readable media are also disclosed.

Systems and methods for selectively augmenting artificial-reality experiences with views of real-world environments

In one embodiment, a method includes receiving a source image and its associated parameters from each of multiple image sources, associating each of the source images with a layer in a range of layers based on the parameters associated with the source images, the range of layers specifying a composition layering order of the source images, generating a corresponding customized distortion mesh for each particular source image in the source images based on the parameters associated with the particular source image and at least a portion of the parameters associated with each of the source images that is associated with any layer preceding a layer associated with the particular source image, modifying each of the source images using the corresponding customized distortion mesh, generating a composite image using the modified source images, and displaying the composite image as a frame in a video.

An image generator is configured to generate display light. A first waveguide is configured to generate wide-field image light from a first portion of the display light. A first outcoupling element of the first waveguide extends to a boundary of the frame to provide the wide-field display to substantially all of the augmented FOV of the user. A second waveguide is configured to generate inset image light from a second portion of the display light received from the image generator.

Wide-field image light and inset image light for head mounted displays

Predictive eye tracking systems and methods for variable focus electronic displays

Systems and methods for performing depth estimation may comprise: an illuminator capable of illuminating a scene from at least a first position and a second position, an image sensor to capture (i) a first image of the scene while the illuminator illuminates the scene from the first position and (ii) a second image of the scene while the illuminator illuminates the scene from the second position, and an image processor to receive the first and second images from the image sensor and estimate a depth of at least one feature that appears in the first and second images. The depth is estimated based on the relative intensity of the first image and the second image, a distance between the first illumination position and the second illumination position, and a position of the at least one feature within at least one of the first and second images.

System and method for depth estimation using multiple illumination sources

An apparatus for acquiring intensity and depth information images may comprise: an image sensing unit having first radiation-sensitive elements and groups of second radiation-sensitive elements in a flat or curved plane having at least two directions, the first radiation-sensitive elements and groups of second radiation-sensitive elements to receive, respectively, an intensity image and a depth information image, at least two groups of second elements extending in each of the two directions of the plane; first micro-lenses, each of which is arranged to convey radiation to a corresponding one of the first elements; and second micro-lenses, each of which is arranged to convey radiation to a corresponding group of the second elements.

Image sensor for depth estimation

A learning technique is provided that learns how to process images by exploiting the spatial and spectral correlations inherent in image data to process and enhance images. Using a training set of input and desired output images, regression coefficients are learned that are optimal for a predefined estimation function that estimates the values at a pixel of the desired output image using a collection of similarly located pixels in the input image. Application of the learned regression coefficients is fast, robust to noise, adapts to the particulars of a dataset, and generalizes to a large variety of applications. The invention enables the use of image sensors with novel color filter array designs that offer expanded capabilities beyond existing sensors and take advantage of typical high pixel counts.

Growing community of inventors

East Palo Alto, CA, United States of America

Steven Paul Lansel