Facebook Technologies, LLC

Meta Platforms Technologies, LLC

Oculus, Vr, LLC

Volga Aksoy

Reza Nourai

Dean Joseph Beeler

Zeyar Htet

Edward Anthony Hutchins

Brian Funt

Paul Louis Pedriana

Yury Anatolievich Petrov

Haomiao Jiang

Curtis Buckoll

In one embodiment, a method includes a step of receiving a geometric representation of a virtual object and a texture atlas, the geometric representation comprising multiple geometric primitives defining a shape of the virtual object, the texture atlas comprises regions each of which is allocated to include shading information of a respective geometric primitive of the plurality of geometric primitives, and the shading information of the respective geometric primitive being scaled down to be smaller than the allocated region so as to create a buffer between the allocated region and adjacent regions of the plurality of regions on the texture atlas. The method further includes steps of identifying, based on a first viewpoint from which to view the virtual object, visible geometric primitives from the plurality of geometric primitives and rendering images of the visible geometric primitives using corresponding shading information included in the texture atlas.

Using a hierarchical organization of a texture atlas to render visible image portions

In one embodiment, a method includes a server generating first shading information for visible portions of objects relative to a first viewpoint, storing the first shading information in a texture atlas, and sending the texture atlas to a client device. The method also includes determining a first subset of the visible portions of the objects for which shading information is to be re-generated and a second subset for which elements of the first shading information are to be reused, generating second shading information for the first subset relative to a second viewpoint, updating the texture atlas to include the second shading information for the first subset and the elements of the first shading information for the second subset, and sending the updated texture atlas to the client device. The updated texture atlas is configured for rendering images of the visible portions of the objects from multiple viewpoints.

Adaptive rate shading using texture atlas

In one embodiment, a method includes the steps of receiving, from a client device, a first viewpoint from which to view a virtual object, the virtual object having a shape defined by multiple geometric primitives, identifying, relative to the first viewpoint, visible geometric primitives from multiple geometric primitives, allocating a region in a texture atlas for each of the visible geometric primitives, generating shading information for each of the visible geometric primitives, storing the shading information of each of the visible geometric primitives in a portion of the allocated region smaller than the allocated region to create a buffer around the portion of the allocated region where the shading information is stored, and sending, to the client device, the texture atlas and a list identifying the visible geometric primitives, the texture atlas being configured for rendering images of the visible geometric primitives from different viewpoints.

Cloud rendering of texture map

A method includes the steps of receiving training data comprising images of an object and associated camera poses from which the images are captured, training, based on the training data, a machine-learning model to take as input a given viewpoint and synthesize an image of a virtual representation of the object viewed from the given viewpoint, generating, for each of predetermined viewpoints surrounding the virtual representation of the object, a view-dependent image of the object as viewed from that viewpoint using the trained machine-learning model, receiving, from a client device, a desired viewpoint from which to view the virtual representation of the object, selecting one or more of the predetermined viewpoints based on the desired viewpoint, and sending, to the client device, the view-dependent images associated with the selected one or more viewpoints for rendering an output image of the virtual representation of the object viewed from the desired viewpoint.

Latency-resilient cloud rendering

Disclosed herein are related to a system and a method of remotely rendering an image. In one approach, a console device generates an image according to a gaze direction of a user of a head mounted display (HMD). In one aspect, the image includes a first area and a second area disposed along an axis, where the second area is located farther away from a foveated area of the image than the first area. In one aspect, the foveated area corresponds to the gaze direction of the user of the HMD. In one aspect, the console device compresses the image according to the axis, where the second area is compressed at a higher level than the first area. In one aspect, the compressed image is transmitted to the HMD. The HMD may decompress the compressed image according to the axis, and render the decompressed image.

Axis based compression for remote rendering

Disclosed herein includes a system, a method, and a device for rendering an image through adaptive reprojection. A first reprojection can be performed to generate a portion of a first image of a first view of a virtual space at a first rate. An amount of change from the first view to a second view of the virtual space can be determined. A portion of a second image of the second view of the virtual space can be determined to generate through a second reprojection. The second reprojection can be performed at a second rate to the portion of the second image according to the amount of change from the first view to the second view of the virtual space.

Adaptive shading and reprojection

A system can include a device and a graphics processing unit (GPU). The device can be configured to receive a first image from one or more cameras corresponding to a first view and a second image from the one or more cameras corresponding to a second view. The GPU can include a motion estimator and configured to receive the first image and the second image and be configured to receive the first image and the second image. The motion estimator can be configured to determine first disparity offsets for the first image and second disparity offsets for the second image. The device can be configured to generate, for rendering 3D image using the first image and the second image, a first depth buffer for the first image derived from the first disparity offsets and a second depth buffer for the second image derived from the second disparity offsets.

Systems and methods of rendering real world objects using depth information

A method includes a computing system receiving a first-person image that captures an artificial-reality environment from a first-person perspective of a user of a head-mounted device, wherein the first-person image is rendered based on a pose of the head-mounted device. Based on the pose of the head-mounted device, the computing system determines a spectator perspective wherein the spectator perspective and the first-person perspective are co-located in a three-dimensional space and the spectator perspective has a predetermined rotational orientation about a view axis. The computing system generates a spectator image that captures the first-person image from the spectator perspective, and causes the first-person image to be displayed by the head-mounted device and the spectator image to be displayed on a device separate from the head-mounted device.

Systems and methods for generating spectator images of an artificial reality environment

The disclosed computer-implemented method may include (1) receiving a first 2D frame depicting an evolving 3D scene and elements in the evolving 3D scene, (2) receiving a second 2D frame depicting the evolving 3D scene and the elements, (3) deriving 2D motion vectors from the first 2D frame and the second 2D frame that each include an estimated offset from coordinates of an element in the first 2D frame to coordinates of the element in the second 2D frame, (4) receiving depth information for the evolving 3D scene, (5) using the 2D motion vectors and the depth information to extrapolate a synthetic 2D frame, and (6) displaying the synthetic 2D frame to a user. Various other methods, systems, and computer-readable media are also disclosed.

Systems and methods for using depth information to extrapolate two-dimensional images

Systems and methods for using depth information to extrapolate two-dimentional images

A headset (e.g., VR headset or AR headset) displays a three-dimensional (3D) virtual scene and includes a distance element to dynamically adjust a distance between an optics block and an electronic display included in the headset based on a location in the virtual scene where the user is looking. The headset tracks the user's eyes to approximate gaze lines and determines a plane of focus for a frame of the virtual scene as the intersection of the gaze lines. The distance element adjusts a distance between an optics block and an electronic display so that the optics block is focused at the plane of focus, which keeps the user's eyes in a zone of comfort as vergence and accommodation change.

Growing community of inventors

Redwood City, CA, United States of America

Volga Aksoy