Nvidia Corporation

Disney Enterprises, Inc.

Eidgenossische Technische Hochschule Zurich

Jan Novák

Fabrice Pierre Armand Rousselle

Alexander Georg Keller

Thomas Müller

Brian McWilliams

Nikolaus Binder

Thijs Vogels

Fabrice Rousselle

Benedikt Martin Bitterli

Wojciech Krzysztof Jarosz

Wenzel A Jakob

Monte Carlo and quasi-Monte Carlo integration are simple numerical recipes for solving complicated integration problems, such as valuating financial derivatives or synthesizing photorealistic images by light transport simulation. A drawback of a straightforward application of (quasi-)Monte Carlo integration is the relatively slow convergence rate that manifests as high error of Monte Carlo estimators. Neural control variates may be used to reduce error in parametric (quasi-)Monte Carlo integration—providing more accurate solutions in less time. A neural network system has sufficient approximation power for estimating integrals and is efficient to evaluate. The efficiency results from the use of a first neural network that infers the integral of the control variate and using normalizing flows to model a shape of the control variate.

Neural network control variates

A fully-connected neural network may be configured for execution by a processor as a fully-fused neural network by limiting slow global memory accesses to reading and writing inputs to and outputs from the fully-connected neural network. The computational cost of fully-connected neural networks scale quadratically with its width, whereas its memory traffic scales linearly. Modern graphics processing units typically have much greater computational throughput compared with memory bandwidth, so that for narrow, fully-connected neural networks, the linear memory traffic is the bottleneck. The key to improving performance of the fully-connected neural network is to minimize traffic to slow 'global' memory (off-chip memory and high-level caches) and to fully utilize fast on-chip memory (low-level caches, “shared” memory, and registers), which is achieved by the fully-fused approach. A real-time neural radiance caching technique for path-traced global illumination is implemented using the fully-fused neural network for caching scattered radiance components of global illumination.

Fully-fused neural network execution

Monte Carlo and quasi-Monte Carlo integration are simple numerical recipes for solving complicated integration problems, such as valuating financial derivatives or synthesizing photorealistic images by light transport simulation. A drawback of a straightforward application of (quasi-)Monte Carlo integration is the relatively slow convergence rate that manifests as high error of Monte Carlo estimators. Neural control variates may be used to reduce error in parametric (quasi-)Monte Carlo integration&#x2014;providing more accurate solutions in less time. A neural network system has sufficient approximation power for estimating integrals and is efficient to evaluate. The efficiency results from the use of a first neural network that infers the integral of the control variate and using normalizing flows to model a shape of the control variate.

A fully-connected neural network may be configured for execution by a processor as a fully-fused neural network by limiting slow global memory accesses to reading and writing inputs to and outputs from the fully-connected neural network. The computational cost of fully-connected neural networks scale quadratically with its width, whereas its memory traffic scales linearly. Modern graphics processing units typically have much greater computational throughput compared with memory bandwidth, so that for narrow, fully-connected neural networks, the linear memory traffic is the bottleneck. The key to improving performance of the fully-connected neural network is to minimize traffic to slow 'global' memory (off-chip memory and high-level caches) and to fully utilize fast on-chip memory (low-level caches, 'shared' memory, and registers), which is achieved by the fully-fused approach. A real-time neural radiance caching technique for path-traced global illumination is implemented using the fully-fused neural network for caching scattered radiance components of global illumination.

A real-time neural radiance caching technique for path-traced global illumination is implemented using a neural network for caching scattered radiance components of global illumination. The neural (network) radiance cache handles fully dynamic scenes, and makes no assumptions about the camera, lighting, geometry, and materials. In contrast with conventional caching, the data-driven approach sidesteps many difficulties of caching algorithms, such as locating, interpolating, and updating cache points. The neural radiance cache is trained via online learning during rendering. Advantages of the neural radiance cache are noise reduction and real-time performance. Importantly, the runtime overhead and memory footprint of the neural radiance cache are stable and independent of scene complexity.

Real-time neural network radiance caching for path tracing

Supervised machine learning using convolutional neural network (CNN) is applied to denoising images rendered by MC path tracing. The input image data may include pixel color and its variance, as well as a set of auxiliary buffers that encode scene information (e.g., surface normal, albedo, depth, and their corresponding variances). In some embodiments, a CNN directly predicts the final denoised pixel value as a highly non-linear combination of the input features. In some other embodiments, a kernel-prediction neural network uses a CNN to estimate the local weighting kernels, which are used to compute each denoised pixel from its neighbors. In some embodiments, the input image can be decomposed into diffuse and specular components. The diffuse and specular components are then independently preprocessed, filtered, and postprocessed, before recombining them to obtain a final denoised image.

Kernel-predicting convolutional neural networks for denoising

Systems, methods and articles of manufacture for rendering three-dimensional virtual environments using reversible jumps are disclosed herein. In one embodiment, mappings from random numbers to light paths are modeled as an explicit iterative random walk. Inverses of path construction techniques are employed to turn light transport paths back into the random numbers that produced them. In particular, such inverses may be used to extend the Multiplexed Metropolis Light Transport (MMLT) technique to perform path-invariant perturbations that produce a new path sample using a different path construction technique but preserve the path's geometry. To render an image, a rendering application in one embodiment may trace light paths through a virtual scene, with some path samples being generated by probabilistically selecting one or more techniques through technique perturbation and using inverses of the selected technique(s) to invert existing path(s), and with new paths being obtained by mutating or perturbing existing paths.

Informed choices in primary sample space for light transport simulation

The disclosure provides an approach for denoising (also referred to as 'filtering') rendered images. In one embodiment, a denoising application takes as input rendered images and feature buffers that encode image information such as surface positions, surface depths, surface normals, surface albedos, and distances to the camera. For each pixel in a received image, the denoising application performs a first-order regression in a predefined neighborhood of the pixel to find a linear combination of pixel features that fits pixel colors in the predefined neighborhood. In such a first-order regression, the local regression weight of each pixel in the neighborhood may be determined using a metric which computes distances based on color values in patches around pixels being compared. In another embodiment, collaborative filtering may be performed in which filtered output from the first-order regression in each neighborhood is averaged with filtered output from overlapping neighborhoods to obtain a final output.

Growing community of inventors

Meilen, Switzerland

Jan Novák