Facebook Technologies, LLC

Meta Platforms Technologies, LLC

Nvidia Corporation

Anton S Kaplanyan

Warren Andrew Hunt

Anjul Patney

Alexander Nankervis

Michael Mara

Douglas Robert Lanman

Alexander Jobe Fix

Lei Xiao

Matthew Steven Chapman

Todd Goodall

Marco Salvi

Aaron Eliot Lefohn

Christopher Ryan Wyman

Jamorn Sriwasansak

Thomas Sebastian Leimkuhler

Mikhail Okunev

Dejan Azinovic

Matthias Niessner

Tzu-Mao Li

Christoph Hermann Schied

David Patrick Luebke

Timo Oskari Aila

John Matthew Burgess

Joohwan Kim

Chakravarty Reddy Alla Chaitanya

Jiahao Lin

Christoph Herman Schied

Salah Eddine Nouri

Anton Sochenov

Christoph H Schied

Nir Benty

In one embodiment, a method includes determining characteristics of one or more areas in an image by analyzing pixels in the image, computing a sampling density for each of the one or more areas in the image based on the characteristics of the one or more areas, generating samples corresponding to the image by sampling pixels in each of the one or more areas according to the associated sampling density, and providing the samples to a machine-learning model as an input, where the machine-learning model is configured to reconstruct the image by processing the samples.

Adaptive sampling of images

In one embodiment, a method includes accessing a pair of stereo images for a scene, where each image of the pair of stereo images has incomplete pixel information and k channels, stacking the pair of stereo images to form a stacked input image with 2k channels, processing the stacked input image using a machine-learning model to generate a stacked output image with 2k channels, and separating the stacked output image with 2k channels into a pair of reconstructed stereo images for the scene, where each image of the pair of reconstructed stereo images has complete pixel information and k channels.

Processing stereo images with a machine-learning model

In one embodiment, a computing system accesses a three-dimensional (3D) model of an environment, the 3D model comprising a virtual representation of an object in the environment. The computing system accesses an image of the object captured by a camera from a camera pose. The computing system accesses light source parameters associated with a virtual representation of a light source in the environment. The computing system renders, using the 3D model, pixels associated with the virtual representation of the object based on the light source parameters, the pixels being rendered from a virtual perspective corresponding to the camera pose. The computing system determines updated light source parameters based on a comparison of the rendered pixels to corresponding pixels located in the image of the object.

Inverse path tracing for material and lighting estimation

In one embodiment, a method includes accessing first-resolution images corresponding to frames of a video, computing a motion vector based on a first-resolution image of a first frame in the video and a first-resolution image of a second frame in the video, generating a second-resolution warped image associated with the second frame by using the motion vector to warp a second-resolution reconstructed image associated with the first frame, generating a second-resolution intermediate image associated with the second frame based on the first-resolution image associated with the second frame, computing adjustment parameters by processing the first-resolution image associated with the second frame and the second-resolution warped image associated with the second frame using a machine-learning model, and adjusting pixels of the second-resolution intermediate image associated with the second frame based on the adjustment parameters to reconstruct a second-resolution reconstructed image associated with the second frame.

Enhancing the resolution of a video stream

Embodiments described herein pertain to a machine-learning approach for shading. A system may determine a number of pixels associated with a viewpoint of a viewer. The system may determine, for each of the pixels, (1) a view direction based on the viewpoint and a pixel position of that pixel and (2) and a surface orientation of a surface visible to that pixel. The system may generate, using a first machine-learning model, a latent space representation of ambient lighting information associated with the pixels based on respective view directions and surface orientations. The system may determine color values for the pixels by processing the latent space representation of ambient lighting information using a second machine-learning model.

Systems and methods for graphics rendering based on machine learning

In one embodiment, a computing system may access a video including a first frame and a second frame. The computing system may determine first sampling locations for the first frame and determine second sampling locations for the second frame by transforming the first sampling locations to the second frame according to an optical flow between the first frame and the second frame. The computing system may select a subset of the second sampling locations based on a comparison between pixels in the first frame corresponding to the first sampling locations and pixels in the second frame corresponding to the second sampling locations. The computing system may define one or more rejection areas in the second frame based on the subset of the second sampling locations to determine third sampling locations in areas outside of the rejection areas. The computing system may generate a sample of the video.

Efficient motion-compensated spatiotemporal sampling

In one embodiment, a computing system may receive a video including a sequence of frames. The computing system may access a three-dimensional mask that specifies pixel-sampling locations, the three-dimensional mask having a first dimension and a second dimension corresponding to a spatial domain and a third dimension corresponding to a temporal domain. Blue noise property may be present in the pixel-sampling locations that are associated with each of a plurality of two-dimensional spatial slices of the three-dimensional mask in the spatial domain and the pixel-sampling locations that are associated with each of a plurality of one-dimensional temporal slices of the three-dimensional mask in the temporal domain. The computing system may generate a sample of the video by sampling the sequence of frames using the three-dimensional mask.

Blue noise mask for video sampling

In one embodiment, a method includes receiving a first frame associated with a first time and one or more second frames of a video having a resolution lower than a target resolution, wherein each second frame is associated with a second time prior to the first time, generating a first feature map for the first frame and one or more second feature maps for the one or more second frames, up-sampling the first feature map and the one or more second feature maps to the target resolution, warping each of the up-sampled second feature maps according to a motion estimation between the associated second time and the first time, and generating a reconstructed frame having the target resolution corresponding to the first frame by using a machine-learning model to process the up-sampled first feature map and the one or more up-sampled and warped second feature maps.

Neural super-sampling for real-time rendering

In one embodiment, a computing system may determine an orientation in a three-dimensional (3D) space and generate a plurality of coordinates in the 3D space based on the determined orientation. The system may access pre-determined ray trajectory definitions associated with the plurality of coordinates. The system may determine visibility information of one or more objects defined within the 3D space by projecting rays through the plurality of coordinates, wherein trajectories of the rays from the plurality of coordinates are determined based on the pre-determined ray trajectory definitions. The system may then generate an image of the one or more objects based on the determined visibility information of the one or more objects.

Systems and methods for rendering optical distortion effects

A method, computer readable medium, and system are disclosed for performing spatiotemporal filtering. The method includes identifying image data to be rendered, reconstructing the image data to create reconstructed image data, utilizing a filter including a neural network having one or more skip connections and one or more recurrent layers, and returning the reconstructed image data.

Filtering image data using a neural network

In one embodiment, a computing system may receive current eye-tracking data associated with a user of a head-mounted display. The system may dynamically adjust a focal length of the head-mounted display based on the current eye-tracking data. The system may generate an in-focus image of a scene and a corresponding depth map of the scene. The system may generate a circle-of-confusion map for the scene based on the depth map. The circle-of-confusion map encodes a desired focal surface in the scene. The system may generate, using a machine-learning model, an output image with a synthesized defocus-blur effect by processing the in-focus image, the corresponding depth map, and the circle-of-confusion map of the scene. The system may display the output image with the synthesized defocus-blur effect to the user via the head-mounted display having the adjusted focal length.

Systems and methods for generating defocus blur effects

In one embodiment, a system may access a training sample that includes training images and corresponding training depth maps of a scene, with the training images being associated with different predetermined viewpoints of the scene. The system may generate elemental images of the scene by processing the training images and the training depth maps using a machine-learning model. The elemental images are associated with more viewpoints of the scene than the predetermined viewpoints associated with the training images. The system may update the machine-learning model based on a comparison between the generated elemental images of the scene and target elemental images that are each associated with a predetermined viewpoint. The updated machine-learning model is configured to generate elemental images of a scene of interest based on input images and corresponding depth maps of the scene of interest from different viewpoints.

Systems and methods utilizing a machine learning model for generating defocus blur effects

In one embodiment, a computing system may determine a first orientation of a viewer in a three-dimensional (3D) space based on first sensor data associated with a first time. The system may render one or more first lines of pixels based on the first orientation of the viewer and display the one or more first lines. The system may determine a second orientation of the viewer in the 3D space based on second sensor data associated with a second time that is subsequent to the first time. The system may render one or more second lines of pixels based on the second orientation of the viewer and display the one or more second lines of pixels. The one or more second lines of pixels associated with the second orientation are displayed concurrently with the one or more first lines of pixels associated with the first orientation.

Systems and methods for reducing rendering latency

In one embodiment, a computing system configured to generate a current frame may access a current sample dataset having incomplete pixel information of a current frame in a sequence of frames. The system may access a previous frame in the sequence of frames with complete pixel information. The system may further access a motion representation indicating pixel relationships between the current frame and the previous frame. The previous frame may then be transformed according to the motion representation. The system may generate the current frame having complete pixel information by processing the current sample dataset and the transformed previous frame using a first machine-learning model.

Neural reconstruction of sequential frames

Embodiments disclosed herein relate to a graphics processing chip for rendering computer graphics. The graphics processing chip may include a controller configured to manage operations of the graphics processing chip in accordance with a graphics-rendering pipeline. The operations may include geometry-processing operations, rasterization operations, and shading operations. The chip may further include programmable memory components configured to store a machine-learning model configured to perform at least a portion of the shading operations. The chip may also include a plurality of processing units configured to be selectively used to perform the shading operations in accordance with the machine-learning model. The chip may also include at least one output memory configured to store image data generated using the shading operations.

Growing community of inventors

Redmond, WA, United States of America

Anton S Kaplanyan