Disney Enterprises, Inc.

Pixar

Eth Zurich (eidgenossische Technische Hochschule Zurich)

Disnev Enterprises, Inc.

Fabrice Rousselle

Brian McWilliams

Thijs Vogels

Mark Joseph Meyer

Jan Novak

Alex Harvill

Jan Novák

A modular architecture is provided for denoising Monte Carlo renderings using neural networks. The temporal approach extracts and combines feature representations from neighboring frames rather than building a temporal context using recurrent connections. A multiscale architecture includes separate single-frame or temporal denoising modules for individual scales, and one or more scale compositor neural networks configured to adaptively blend individual scales. An error-predicting module is configured to produce adaptive sampling maps for a renderer to achieve more uniform residual noise distribution. An asymmetric loss function may be used for training the neural networks, which can provide control over the variance-bias trade-off during denoising.

Temporal techniques of denoising Monte Carlo renderings using neural networks

Supervised machine learning using neural networks is applied to denoising images rendered by MC path tracing. Specialization of neural networks may be achieved by using a modular design that allows reusing trained components in different networks and facilitates easy debugging and incremental building of complex structures. Specialization may also be achieved by using progressive neural networks. In some embodiments, training of a neural-network based denoiser may use importance sampling, where more challenging patches or patches including areas of particular interests within a training dataset are selected with higher probabilities than others. In some other embodiments, generative adversarial networks (GANs) may be used for training a machine-learning based denoiser as an alternative to using pre-defined loss functions.

Denoising Monte Carlo renderings using progressive neural networks

Supervised machine learning using convolutional neural network (CNN) is applied to denoising images rendered by MC path tracing. The input image data may include pixel color and its variance, as well as a set of auxiliary buffers that encode scene information (e.g., surface normal, albedo, depth, and their corresponding variances). In some embodiments, a CNN directly predicts the final denoised pixel value as a highly non-linear combination of the input features. In some other embodiments, a kernel-prediction neural network uses a CNN to estimate the local weighting kernels, which are used to compute each denoised pixel from its neighbors. In some embodiments, the input image can be decomposed into diffuse and specular components. The diffuse and specular components are then independently preprocessed, filtered, and postprocessed, before recombining them to obtain a final denoised image.

Kernel-predicting convolutional neural networks for denoising

Denoising Monte Carlo renderings using machine learning with importance sampling

Adaptive sampling in Monte Carlo renderings using error-predicting neural networks

Denoising Monte Carlo renderings using neural networks with asymmetric loss

Multi-scale architecture of denoising monte carlo renderings using neural networks

Denoising monte carlo renderings using progressive neural networks

Denoising Monte Carlo renderings using generative adversarial neural networks

Growing community of inventors

Ostermundingen, Switzerland

Fabrice Rousselle