Meta Platforms Technologies, LLC

Google Inc.

Facebook Technologies, LLC

Haomiao Jiang

Behnam Bastani

Tianxin Ning

Brian Funt

Reza Nourai

Zhang Jia

Steven Paul Lansel

Carlin Vieri

Volga Aksoy

Linghui Rao

Weihua Gao

Eric Turner

Agnes Lee

Rohit Rao Padebettu

Curtis Buckoll

Todd Douglas Keeler

Damien Saint-Macary

Xiaobo Tian

Yongmin Park

Grant Kaijuin Yang

Sylvain Vignaud

In particular embodiments, a computing system may receive receiving a first image frame of a sequence of image frames to be shown on a display having a plurality of backlight zones. The computing system may compute backlight unit statistics of the first image frame. The backlight unit statistics may represent grayscale levels for the plurality of backlight zones. The computing system may compute a global dimming gain for adjusting color values and backlight unit intensity of a second image frame of the sequence of image frames based on the backlight unit statistics of the first image frame. The computing system may adjust, using the global dimming gain, the color values and the backlight unit intensity of the second image frame.

Systems and methods for performing global dimming in artificial reality systems

A method for providing imagery to a user on a display includes receiving eye tracking data. The method also includes determining a current gaze location and a relative distance between the current gaze location and an edge of the display using the eye tracking data. The method also includes defining a first tile centered at the current gaze location and multiple tiles that surround the first tile using the current gaze location and the relative distance between the current gaze location and the edge of the display. The method includes providing a foveated rendered image using the first tile and the multiple tiles.

Foveated rendering using eye motion

In particular embodiments, the disclosure provides a method comprising: rendering, on a graphics processing unit (GPU), a low-resolution image associated with a scene, the low-resolution image having a resolution that is lower than a target resolution; transmitting a version of the low-resolution image to a neural accelerator; processing, on the neural accelerator, the version of the low-resolution image using a trained machine-learning model, thereby outputting a plurality of control parameters; transmitting the control parameters from the neural accelerator to the GPU; processing, on the GPU, the low-resolution image and the control parameters to construct a high-resolution image having the target resolution, wherein the GPU is programmed to determine a plurality of pixel weights for performing an interpolation using the control parameters; and outputting the high-resolution image.

Split machine-learning assisted graphics rendering pipeline

Disclosed herein are related to a system and a method for providing an artificial reality. In one aspect, a system includes a shared physical memory and a first processor having access to the shared physical memory. In one aspect, the first processor performs, during a first time period, a first rendering process to generate a first image frame of a first view of an artificial reality. In one aspect, the first processor performs, during a second time period, a second rendering process to generate a second image frame of a second view of the artificial reality. In one aspect, the system includes a second processor including a neural network and having access to the shared physical memory. In one aspect, the second processor performs, during a third time period overlapping a portion of the second time period, an image enhancing process on the first image frame.

Pipeline with processor including neural network for rendering artificial reality

In one embodiment, a method includes a server generating first shading information for visible portions of objects relative to a first viewpoint, storing the first shading information in a texture atlas, and sending the texture atlas to a client device. The method also includes determining a first subset of the visible portions of the objects for which shading information is to be re-generated and a second subset for which elements of the first shading information are to be reused, generating second shading information for the first subset relative to a second viewpoint, updating the texture atlas to include the second shading information for the first subset and the elements of the first shading information for the second subset, and sending the updated texture atlas to the client device. The updated texture atlas is configured for rendering images of the visible portions of the objects from multiple viewpoints.

Adaptive rate shading using texture atlas

Embodiments of the present disclosure relate to a display assembly with color calibration for improving accuracy of displayed colors under thermal shifts. A data processing circuit of the display assembly determines color compensation coefficients corresponding to an operating temperature of a display panel, each color compensation coefficient accounting for a thermal shift of a respective color component at the operating temperature. The data processing circuit applies the color compensation coefficients to color transform elements corresponding to an initial temperature to determine a corrected version of the color transform elements corresponding to the operating temperature. The data processing circuit processes pixel values of color components in a first format associated with the display panel at the operating temperature by applying the corrected version of the color transform elements to generate corrected pixel values of the color components in a second format for presentation on the display panel at the operating temperature.

Real-time color conversion in display panels under thermal shifts

A method for providing imagery to a user on a display includes receiving head tracking data and using the head tracking data to determine a change in an orientation of a user's head. The method also includes blending pixels of a previous frame with corresponding pixels of a current frame. The corresponding pixels of the current frame are identified using the change in the orientation of the user's head. The method also includes providing foveated imagery to the user on the display using the blended pixels and multiple tiles.

Systems and methods for temporal anti-aliasing

A method for providing imagery to a user on a display includes receiving eye tracking data. The method also includes determining a gaze location on the display and at least one of a confidence factor of the gaze location, or a speed of the change of the gaze location using the eye tracking data. The method also includes establishing multiple tiles using the gaze location and at least one of the confidence factor or the speed of the change of the gaze location. The method also includes providing a foveated rendered image using the multiple tiles.

A display device, such as a head mounted device (HMD), displays a virtual scene. The display device includes a motion tracker for detecting rotation and/or translation of the display device. The display device also includes a processor that is configured to determine, in response to the detected, an orientation of the display device relative to a plurality of world-aligned viewing frustums that are stationary relative to the virtual scene. The processor is also configured to identify a set of those world-aligned viewing frustums that overlap with an output field of view. The processor is further configured to render pixels of the set of those world-aligned viewing frustums that overlap with an output field of view and upsample the rendered pixels to generate values of display pixels for presentation by the display device.

Phase aligned foveated rendering

Systems and methods for performing foveated rendering are provided. An example system and method may warp a 3D scene based on a fixation point. The system and method may also render the warped 3D scene to generate a first image. The system and method may also unwarp the first image to generate a second image. For example, the first image may have fewer pixels than the second image.

Smoothly varying foveated rendering

A processing unit is configured to render first pixels representative of a high-acuity region in the image and second pixels representative of a low-acuity region in the image. A shaper is configured to reorganize the first pixels based on at least one dimension of the low-acuity region. A multiplexer is configured to multiplex the reorganized first pixels and the second pixels to form a display stream. An encoder is configured to compress the display stream for transmission to a display. A decoder configured to decompress the display stream. A demultiplexer is configured to demultiplex the first pixels and the second pixels. Another processing unit is configured to blend the first pixels and the second pixels to form blended pixel values representative of the image for presentation on a screen.

Growing community of inventors

Sunnyvale, CA, United States of America

Haomiao Jiang