Tandent Vision Science, Inc.

Adobe, Inc.

Disney Enterprises, Inc.

UniversitÈ Laval

Tandent Vision Sciencem, Inc.

Jean-Francois Lalonde

Kalyan Krishna Sunkavalli

Sunil Sharadchandra Hadap

Iain Matthews

Andrew Neil Stein

Yannick Hold-Geoffroy

Nathan Aaron Carr

Casey Arthur Smith

Richard Mark Friedhoff

Bruce Allen Maxwell

Jonathan Eisenmann

Mathieu Garon

Jinsong Zhang

Patrick Buehler

Christian Gagne

This disclosure relates to methods, non-transitory computer readable media, and systems that can render a virtual object in a digital image by using a source-specific-lighting-estimation-neural network to generate three-dimensional ('3D') lighting parameters specific to a light source illuminating the digital image. To generate such source-specific-lighting parameters, for instance, the disclosed systems utilize a compact source-specific-lighting-estimation-neural network comprising both common network layers and network layers specific to different lighting parameters. In some embodiments, the disclosed systems further train such a source-specific-lighting-estimation-neural network to accurately estimate spatially varying lighting in a digital image based on comparisons of predicted environment maps from a differentiable-projection layer with ground-truth-environment maps.

Dynamically estimating light-source-specific parameters for digital images using a neural network

Methods and systems are provided for determining high-dynamic range lighting parameters for input low-dynamic range images. A neural network system can be trained to estimate high-dynamic range lighting parameters for input low-dynamic range images. The high-dynamic range lighting parameters can be based on sky color, sky turbidity, sun color, sun shape, and sun position. Such input low-dynamic range images can be low-dynamic range panorama images or low-dynamic range standard images. Such a neural network system can apply the estimates high-dynamic range lighting parameters to objects added to the low-dynamic range images.

Learning from estimated high-dynamic range all weather lighting parameters

This disclosure relates to methods, non-transitory computer readable media, and systems that use a local-lighting-estimation-neural network to render a virtual object in a digital scene by using a local-lighting-estimation-neural network to analyze both global and local features of the digital scene and generate location-specific-lighting parameters for a designated position within the digital scene. For example, the disclosed systems extract and combine such global and local features from a digital scene using global network layers and local network layers of the local-lighting-estimation-neural network. In certain implementations, the disclosed systems can generate location-specific-lighting parameters using a neural-network architecture that combines global and local feature vectors to spatially vary lighting for different positions within a digital scene.

Dynamically estimating lighting parameters for positions within augmented-reality scenes based on global and local features

Methods and systems for estimating HDR sky light probes for outdoor images are disclosed. A precaptured sky light probe database is leveraged. The database includes a plurality of HDR sky light probes captured under a plurality of different illumination conditions. A HDR sky light probe is estimated from an outdoor image by fitting a three dimensional model to an object of interest in the image and solving an inverse optimization lighting problem for the 3D model where the space of possible HDR sky light probes is constrained by the HDR sky light probes of the database.

Systems and methods for estimating sky light probes for outdoor images

Methods and systems for predicting light probes for outdoor images are disclosed. A light probe database is created to learn a mapping from the outdoor image's features to predicted outdoor light probe illumination parameters. The database includes a plurality of images, image features for each of the plurality of images, and a captured light probe for each of the plurality of images. A light probe illumination model based on a sun model and sky model is fitted to the captured light probes. The light probe for the outdoor image may be predicted based on the database dataset and fitted light probe models.

Predicting a light probe for an outdoor image

In a first exemplary embodiment, an automated, computerized method is provided for processing an image. The method includes the steps of providing an image file depicting an image defined by image locations, in a computer memory, generating a bi-illuminant chromaticity plane in a log color space for representing the image locations of the image in a log-chromaticity representation for the image, providing a set of estimates for an orientation of the bi-illuminant chromaticity plane and calculating a single orientation for each one of the image locations as a function of the set of estimates for an orientation.

Spatially varying log-chromaticity normals for use in an image process

In a first exemplary embodiment of the present invention, an automated, computerized method is provided for processing an image. According to a feature of the present invention, the method comprises the steps of providing an image represented in a spatial plane, organizing spatio-spectral information for the image in a matrix equation expressed by normal and tangential constraints determined as a function of a spatial orientation of selected image locations, relative to the spatial plane of the image and utilizing the matrix equation to solve for an intrinsic image corresponding to the image.

Oriented, spatio-spectral illumination constraints for use in an image progress

In a first exemplary embodiment of the present invention, an automated, computerized method is provided for processing an image. According to a feature of the present invention, the method comprises the steps of providing an image file depicting an image defined by image locations, in a computer memory, generating a bi-illuminant chromaticity plane in a log color space for representing the image locations of the image in a log-chromaticity representation for the image, providing a set of estimates for the orientation of the bi-illuminant chromaticity plane and calculating an orientation for each one of the image locations as a function of the set of estimates for the orientation.

Log-chromaticity clustering pipeline for use in an image process

In a first exemplary embodiment of the present invention, an automated, computerized method is provided for processing an image. According to a feature of the present invention, the method comprises the steps of providing an image file depicting an image defined by image locations, in a computer memory, generating a bi-illuminant chromaticity plane in a log color space for representing the image locations of the image in a log-chromaticity representation for the image, providing a set of estimates for an orientation of the bi-illuminant chromaticity plane and calculating a single orientation for each one of the image locations as a function of the set of estimates for an orientation.

In a first exemplary embodiment of the present invention, an automated, computerized method is provided for processing an image. According to a feature of the present invention, the method comprises the steps of providing an image file depicting an image defined by image locations, in a computer memory, generating a bi-illuminant chromaticity plane in a log color space for representing the image locations of the image in a log-chromaticity representation for the image, providing an estimate for an orientation of the bi-illuminant chromaticity plane and optimizing the estimate of an orientation as a function of a minimization of a weighted entropy.

Growing community of inventors

Pittsburgh, PA, United States of America

Jean-Francois Lalonde