Google Inc.

Meta Platforms Technologies, LLC

Facebook Technologies, LLC

X Development LLC

Qualcomm Mems Technologies, Inc.

Behnam Bastani

Mary Lou Jepsen

Haomiao Jiang

Carlin Vieri

Nicholas Charles Loomis

Andrew John Ouderkirk

Gang Lu

Andrei Stefan Kazmierski

Xiaoguang Wang

Eric Turner

Tianxin Ning

Nicholas Daniel Trail

Pasi Saarikko

Mark B Spitzer

Hee Yoon Lee

Adam Eales Norton

Reza Nourai

Peyman Milanfar

Sharvil Shailesh Talati

Belle Fu

Philip Edwin Watson

Colin Andrew Braley

Steven Paul Lansel

Mohsen Shahmohammadi

Brian Funt

Stuart C B Abercrombie

Pey Lung Fu

Hung-Yu Chen

James Randal Hughes

Damien Saint-Macary

Lu Lu

Robin Sharma

Katherine Marie Smyth

Karol Constantine Hatzilias

Alok Govil

Byron Taylor

Eliezer Peli

John M Cooke

Nagi H Elabbasi

Serge J A Bierhuizen

Stephen Duane Dorow

Rohit Rao Padebettu

Todd Douglas Keeler

Matthew Robert Fulghum

Curtis Buckoll

Yachi Lee

Ya-Chi Lee

Eddy Giing-Li Chen

Shenyi Tai

Pouya Bastani

Hagar Edelstain

Yufeng Zhu

Matthew James Hancock

Grant Kaijuin Yang

Sylvain Vignaud

A method for providing imagery to a user on a display includes receiving eye tracking data. The method also includes determining a current gaze location and a relative distance between the current gaze location and an edge of the display using the eye tracking data. The method also includes defining a first tile centered at the current gaze location and multiple tiles that surround the first tile using the current gaze location and the relative distance between the current gaze location and the edge of the display. The method includes providing a foveated rendered image using the first tile and the multiple tiles.

Foveated rendering using eye motion

In one embodiment, a method includes accessing a first image corresponding to a first frame of a video stream, where the first image has complete pixel information, rendering a provisional image corresponding to a second frame of the video stream subsequent to the first frame, where the provisional image has a first area with complete pixel information and a second area with incomplete pixel information, generating a predicted image corresponding to the second frame by re-projecting at least an area of the first image according to one or more warping parameters, and generating a second image corresponding to the second frame by compositing the rendered provisional image and the predicted image.

Temporal foveated rendering

In particular embodiments, the disclosure provides a method comprising: rendering, on a graphics processing unit (GPU), a low-resolution image associated with a scene, the low-resolution image having a resolution that is lower than a target resolution; transmitting a version of the low-resolution image to a neural accelerator; processing, on the neural accelerator, the version of the low-resolution image using a trained machine-learning model, thereby outputting a plurality of control parameters; transmitting the control parameters from the neural accelerator to the GPU; processing, on the GPU, the low-resolution image and the control parameters to construct a high-resolution image having the target resolution, wherein the GPU is programmed to determine a plurality of pixel weights for performing an interpolation using the control parameters; and outputting the high-resolution image.

Split machine-learning assisted graphics rendering pipeline

Disclosed herein are related to a system and a method for providing an artificial reality. In one aspect, a system includes a shared physical memory and a first processor having access to the shared physical memory. In one aspect, the first processor performs, during a first time period, a first rendering process to generate a first image frame of a first view of an artificial reality. In one aspect, the first processor performs, during a second time period, a second rendering process to generate a second image frame of a second view of the artificial reality. In one aspect, the system includes a second processor including a neural network and having access to the shared physical memory. In one aspect, the second processor performs, during a third time period overlapping a portion of the second time period, an image enhancing process on the first image frame.

Pipeline with processor including neural network for rendering artificial reality

In one embodiment, the disclosure provides a computer-implemented method for Progressive Subsampled Transmission of image data. In one embodiment, a source computer may: generate a first down-sampled frame by sampling an input image according to a first sampling pattern; generate a first encoded down-sampled frame; transmit the first encoded down-sampled frame to a recipient device to cause the recipient device to display/use a first output frame generated by decoding and up-sampling the first encoded down-sampled frame; generate a second down-sampled frame by sampling the input image according to a second sampling pattern; generate a second encoded down-sampled frame; and transmit the second encoded down-sampled frame to the recipient device to cause the recipient device to display/use a second output frame generated based on the first encoded down-sampled frame and the second encoded down-sampled frame and in accordance with the first sampling pattern and the second sampling pattern.

Progressive transmission of detailed image data via video compression of successive subsampled frames

A method for correcting artifacts in artificial reality headsets is provided. The method includes coupling a first image portion into a waveguide in a display for an artificial reality device, provided by a first array of pixels, coupling a second image portion into the waveguide in the display, provided by a second array of pixels, directing the first image portion through an eyebox, directing the second image portion through the eyebox, and tiling, in a user's retina, the first image portion and the second image portion to form an image having an extended field of view that includes an overlapping area comprising light provided by the first array of pixels and the second array of pixels. A system including a memory storing instructions and a processor to execute the instructions to cause the system to perform the above method are also provided.

Correcting artifacts in tiled display assemblies for artificial reality headsets

Disclosed herein are related to systems and methods for providing artificial reality. In one aspect, a console receives feedback information indicative of a first completion time from a head wearable display (HWD). The HWD may complete generating a first image frame at the first completion time. The HWD may display the first image frame. In one aspect, the console compares a display time, at which the first image frame is displayed by the HWD, and the first completion time. In one aspect, the console adjusts, according to the comparison, image processing, to adjust a second completion time, at which the console completes generating a second image frame. In one aspect, the console generates, via the adjusted image processing, the second image frame to provide to the HWD.

Adaptive rate control for artificial reality

Disclosed herein includes a system, a method, and a device for prioritizing packet retransmission. A transmitting device can insert, for each packet of a plurality of packets of a video frame, a sequence number indicative of an order of the corresponding packet among the plurality of packets, into a header of the corresponding packet according to an application layer protocol. The transmitting device can transmit to the receiving device, at a first level of priority, the plurality of packets including the corresponding inserted sequence numbers. The transmitting device can receive an identification of one or more packets to be retransmitted to the receiving device, the identification based on at least the inserted sequence numbers of the one or more packets. The transmitting device can retransmit the one or more packets at a second level of priority that is higher than the first level of priority.

Systems and methods for prioritizing packet retransmission

Disclosed herein are related to a device and a method of remotely rendering an image. In one approach, a device divides an image of an artificial reality space into a plurality of slices. In one approach, the device encodes a first slice of the plurality of slices. In one approach, the device encodes a portion of a second slice of the plurality of slices, while the device encodes a portion of the first slice. In one approach, the device transmits the encoded first slice of the plurality of slices to a head wearable display. In one approach, the device transmits the encoded second slice of the plurality of slices to the head wearable display, while the device transmits a portion of the encoded first slice to the head wearable display.

Sliced encoding and decoding for remote rendering

Disclosed herein includes a system, a method, and a device for reducing latency and jitter by reserving a transmission duration for a transmit opportunity. A device can compress a first slice of a plurality of slices of a video frame to obtain a size of the first slice after compression. The device can estimate a size of the video frame after compression, according to the size of the first slice after compression. The device can generate a request to send (RTS) to reserve, a transmission channel for a transmission duration corresponding to the size of the video frame after compression. The device can broadcast the RTS to one or more other devices.

Systems and methods for reducing WiFi latency using transmit opportunity and duration

A method for providing imagery to a user on a display includes receiving head tracking data and using the head tracking data to determine a change in an orientation of a user's head. The method also includes blending pixels of a previous frame with corresponding pixels of a current frame. The corresponding pixels of the current frame are identified using the change in the orientation of the user's head. The method also includes providing foveated imagery to the user on the display using the blended pixels and multiple tiles.

Systems and methods for temporal anti-aliasing

A near-eye-display (NED) includes a tracking system and a waveguide assembly. The waveguide assembly includes an infrared (IR) light source and an output waveguide. The output waveguide includes at least a decoupling element that outcouples the IR light emitted by the IR light source to form the structured light pattern. The structured light pattern is projected toward one or more regions of a user's face, for example, the user's eyes. The structured light pattern is reflected off the one or more regions of the user's face and captured by the tracking system. The tracking system can determine tracking information such as eye tracking information as well as face tracking information based on the captured reflected structured light pattern.

Waveguide display with structured light for eye and face tracking

A method for providing imagery to a user on a display includes receiving eye tracking data. The method also includes determining a gaze location on the display and at least one of a confidence factor of the gaze location, or a speed of the change of the gaze location using the eye tracking data. The method also includes establishing multiple tiles using the gaze location and at least one of the confidence factor or the speed of the change of the gaze location. The method also includes providing a foveated rendered image using the multiple tiles.

Growing community of inventors

San Jose, CA, United States of America

Behnam Bastani