Nvidia Corporation

Facebook Technologies, LLC

Meta Platforms Technologies, LLC

Anjul Patney

Aaron Eliot Lefohn

Marco Salvi

Anton S Kaplanyan

Christopher Ryan Wyman

Todd Goodall

David Patrick Luebke

Carl Jacob Munkberg

Morgan McGuire

Jon Niklas Theodor Hasselgren

Qi Sun

Omer Shapira

Donald Lee Brittain

William James Dally

Henry Packard Moreton

Eric Brian Lum

Yury Y Uralsky

Eric Enderton

Kayvon Fatahalian

Yong He

Shaveen Kumar

G Evan Hart, Jr

Jamorn Sriwasansak

Eric Xu

Anton Moor

Yubo Zhang

Jan Kautz

John Matthew Burgess

Michael C Shebanow

Chao Liu

Romain Bachy

Joohwan Kim

Iuri Frosio

Thomas Sebastian Leimkuhler

Hongxu Yin

Yazhou Xing

Amrita Mazumdar

Trisha Lian

Christoph H Schied

Gizem Rufo

Nir Benty

One or more embodiments include receiving one or more frames of a live video captured by a video capturing device, wherein the one or more frames include a current frame that is most-recently captured, identifying a set of reference frames included in the one or more frames based on at least the current frame, wherein each frame in the set of reference frames has a different exposure level relative to the current frame, determining, using one or more neural networks, a set of missing details for one or more regions of the current frame based on the set of reference frames, generating an updated version of the current frame based on the set of details, and outputting the updated version of the current frame in real-time.

Hallucinating details for over-exposed pixels in videos using learned reference frame selection

The technology disclosed herein involves using a machine learning model (e.g., CNN) to expand lower dynamic-range image content (e.g., SDR images) into higher dynamic-range image content (e.g., HDR images). The machine learning model can take as input the lower dynamic-range image and can output multiple expansion maps that are used to make the expanded image appear more natural. The expansion maps may be used by image operators to smooth color banding and to dim overexposed regions or user interface elements in the expanded image. The expanded content (e.g., HDR image content) may then be provided to one or more devices for display or storage.

Remastering lower dynamic range content for higher dynamic range displays

A neural network-based rendering technique increases temporal stability and image fidelity of low sample count path tracing by optimizing a distribution of samples for rendering each image in a sequence. A sample predictor neural network learns spatio-temporal sampling strategies such as placing more samples in dis-occluded regions and tracking specular highlights. Temporal feedback enables a denoiser neural network to boost the effective input sample count and increases temporal stability. The initial uniform sampling step typically present in adaptive sampling algorithms is not needed. The sample predictor and denoiser operate at interactive rates to achieve significantly improved image quality and temporal stability compared with conventional adaptive sampling techniques.

Neural network system with temporal feedback for denoising of rendered sequences

Motivated by the ability of humans to quickly and accurately detect visual artifacts in images, a neural network model is trained to identify and locate visual artifacts in a sequence of rendered images without comparing the sequence of rendered images against a ground truth reference. Examples of visual artifacts include aliasing, blurriness, mosaicking, and overexposure. The neural network model provides a useful fully-automated tool for evaluating the quality of images produced by rendering systems. The neural network model may be trained to evaluate the quality of images for video processing, encoding, and/or compression techniques. In an embodiment, the sequence includes at least four images corresponding to a video or animation.

Detecting visual artifacts in image sequences using a neural network model

In one embodiment, a method includes determining characteristics of one or more areas in an image by analyzing pixels in the image, computing a sampling density for each of the one or more areas in the image based on the characteristics of the one or more areas, generating samples corresponding to the image by sampling pixels in each of the one or more areas according to the associated sampling density, and providing the samples to a machine-learning model as an input, where the machine-learning model is configured to reconstruct the image by processing the samples.

Adaptive sampling of images

In one embodiment, a method includes accessing a pair of stereo images for a scene, where each image of the pair of stereo images has incomplete pixel information and k channels, stacking the pair of stereo images to form a stacked input image with 2k channels, processing the stacked input image using a machine-learning model to generate a stacked output image with 2k channels, and separating the stacked output image with 2k channels into a pair of reconstructed stereo images for the scene, where each image of the pair of reconstructed stereo images has complete pixel information and k channels.

Processing stereo images with a machine-learning model

Neural network system with temporal feedback for adaptive sampling and denoising of rendered sequences

In one embodiment, a computing system may access a video including a first frame and a second frame. The computing system may determine first sampling locations for the first frame and determine second sampling locations for the second frame by transforming the first sampling locations to the second frame according to an optical flow between the first frame and the second frame. The computing system may select a subset of the second sampling locations based on a comparison between pixels in the first frame corresponding to the first sampling locations and pixels in the second frame corresponding to the second sampling locations. The computing system may define one or more rejection areas in the second frame based on the subset of the second sampling locations to determine third sampling locations in areas outside of the rejection areas. The computing system may generate a sample of the video.

Efficient motion-compensated spatiotemporal sampling

In one embodiment, a computing system may receive a video including a sequence of frames. The computing system may access a three-dimensional mask that specifies pixel-sampling locations, the three-dimensional mask having a first dimension and a second dimension corresponding to a spatial domain and a third dimension corresponding to a temporal domain. Blue noise property may be present in the pixel-sampling locations that are associated with each of a plurality of two-dimensional spatial slices of the three-dimensional mask in the spatial domain and the pixel-sampling locations that are associated with each of a plurality of one-dimensional temporal slices of the three-dimensional mask in the temporal domain. The computing system may generate a sample of the video by sampling the sequence of frames using the three-dimensional mask.

Blue noise mask for video sampling

In one embodiment, a method includes projecting a source image onto a surface using a lens approximation component, where the surface is associated with sampling points approximating photoreceptors of an eye, where each sampling point has a corresponding photoreceptor type, sampling color information from the projected source image at the sampling points, where the color information sampled at each sampling point depends on the corresponding photoreceptor type, accessing pooling units approximating retinal ganglion cells (RGCs) of the eye, where each pooling unit is associated with groups of one or more of the sampling points, calculating weighted aggregations of the sampled color information associated with the groups of one or more sampling points associated with each pooling unit, and computing a perception profile for the source image based on the weighted aggregations associated with each of the pooling units.

Calibrated sensitivity model approximating the eye

A method, computer readable medium, and system are disclosed for redirecting a user's movement through a physical space while the user views a virtual environment. A temporary visual suppression event is detected when a user's eyes move relative to the user's head while viewing a virtual scene displayed on a display device, an orientation of the virtual scene relative to the user is modified to direct the user to physically move along a planned path through a virtual environment corresponding to the virtual scene, and the virtual scene is displayed on the display device according to the modified orientation.

Saccadic redirection for virtual reality locomotion

A method, computer readable medium, and system are disclosed for performing spatiotemporal filtering. The method includes the steps of applying, utilizing a processor, a temporal filter of a filtering pipeline to a current image frame, using a temporal reprojection, to obtain a color and auxiliary information for each pixel within the current image frame, providing the auxiliary information for each pixel within the current image frame to one or more subsequent filters of the filtering pipeline, and creating a reconstructed image for the current image frame, utilizing the one or more subsequent filters of the filtering pipeline.

Performing spatiotemporal filtering

A method, computer readable medium, and system are disclosed for computing a path for a user to move along within a physical space while viewing a virtual environment in a virtual reality system. A path for a user to physically move along through a virtual environment is determined based on waypoints and at least one characteristic of the physical environment within which the user is positioned, position data for the user is received indicating whether and how much a current path taken by the user has deviated from the path, and an updated path is computed through the virtual environment based on the waypoints and the at least one characteristic of the physical environment.

Path planning for virtual reality locomotion

Growing community of inventors

Kirkland, WA, United States of America

Anjul Patney