Disney Enterprises, Inc.

Pixar

Eth Zurich (eidgenossische Technische Hochschule Zurich)

Disnev Enterprises, Inc.

Thijs Vogels

Brian McWilliams

Fabrice Rousselle

Mark Joseph Meyer

Jan Novak

Alex Harvill

Jan Novák

Christopher Richard Schroers

Marios Papas

David M Adler

Farnood Salehi

Marco Manzi

Mark A Meyer

Henrik D Dahlberg

Gerhard Röthlin

Andre C Mazzone

Per H Christensen

Certain aspects of the present disclosure provide techniques for adaptive sampling for rendering using deep learning. This includes receiving, at a sampler in a rendering pipeline, a plurality of rendered pixel data, wherein the sampler includes a first machine learning (ML) model. It further includes generating a sampling map for the rendering pipeline using the first ML model and the plurality of rendered pixel data, including predicting a plurality of pixel values in the sampling map based on a generated distribution of pixel values. It further includes rendering an image using the sampler, the sampling map, and a denoiser in the rendering pipeline.

Adaptive sampling using deep learning

A modular architecture is provided for denoising Monte Carlo renderings using neural networks. The temporal approach extracts and combines feature representations from neighboring frames rather than building a temporal context using recurrent connections. A multiscale architecture includes separate single-frame or temporal denoising modules for individual scales, and one or more scale compositor neural networks configured to adaptively blend individual scales. An error-predicting module is configured to produce adaptive sampling maps for a renderer to achieve more uniform residual noise distribution. An asymmetric loss function may be used for training the neural networks, which can provide control over the variance-bias trade-off during denoising.

Temporal techniques of denoising Monte Carlo renderings using neural networks

Supervised machine learning using neural networks is applied to denoising images rendered by MC path tracing. Specialization of neural networks may be achieved by using a modular design that allows reusing trained components in different networks and facilitates easy debugging and incremental building of complex structures. Specialization may also be achieved by using progressive neural networks. In some embodiments, training of a neural-network based denoiser may use importance sampling, where more challenging patches or patches including areas of particular interests within a training dataset are selected with higher probabilities than others. In some other embodiments, generative adversarial networks (GANs) may be used for training a machine-learning based denoiser as an alternative to using pre-defined loss functions.

Denoising Monte Carlo renderings using progressive neural networks

Supervised machine learning using convolutional neural network (CNN) is applied to denoising images rendered by MC path tracing. The input image data may include pixel color and its variance, as well as a set of auxiliary buffers that encode scene information (e.g., surface normal, albedo, depth, and their corresponding variances). In some embodiments, a CNN directly predicts the final denoised pixel value as a highly non-linear combination of the input features. In some other embodiments, a kernel-prediction neural network uses a CNN to estimate the local weighting kernels, which are used to compute each denoised pixel from its neighbors. In some embodiments, the input image can be decomposed into diffuse and specular components. The diffuse and specular components are then independently preprocessed, filtered, and postprocessed, before recombining them to obtain a final denoised image.

Kernel-predicting convolutional neural networks for denoising

Denoising Monte Carlo renderings using machine learning with importance sampling

Adaptive sampling in Monte Carlo renderings using error-predicting neural networks

Denoising Monte Carlo renderings using neural networks with asymmetric loss

Multi-scale architecture of denoising monte carlo renderings using neural networks

Denoising monte carlo renderings using progressive neural networks

Denoising Monte Carlo renderings using generative adversarial neural networks

Growing community of inventors

Lausanne, Switzerland

Thijs Vogels