Adobe, Inc.

Imperial College Innovations Limited

Valentin Deschaintre

Milos Hasan

Paul Augusto Guerrero

Yiwei Hu

Tamy Boubekeur

Abhijeet Ghosh

Julien Philip

Yiming Lin

Arthur Jules Martin Roullier

Shouchang Guo

Jerome Derel

Paul Parneix

Methods and systems disclosed herein relate generally to radiance field gradient scaling for unbiased near-camera training. In a method, a processing device accesses an input image of a three-dimensional environment comprising a plurality of pixels, each pixel comprising a pixel color. The processing device determines a camera location based on the input image and a ray from the camera location in a direction of a pixel. The processing device integrates sampled information from a volumetric representation along the ray from the camera location to obtain an integrated color. The processing device trains a machine learning model configured to predict a density and a color, comprising minimizing a loss function using a scaling factor that is determined based on a distance between the camera location and a point along the ray. The processing device outputs the trained ML model for use in rendering an output image.

Radiance field gradient scaling for unbiased near-camera training

Systems and methods for image processing are described. Embodiments of the present disclosure include an image processing apparatus configured to efficiently perform texture synthesis (e.g., increase the size of, or extend, texture in an input image while preserving a natural appearance of the synthesized texture pattern in the modified output image). In some aspects, the image processing apparatus implements an attention mechanism with a multi-stage attention model where different stages (e.g., different transformer blocks) progressively refine image feature patch mapping at different scales, while utilizing repetitive patterns in texture images to enable network generalization. One or more embodiments of the disclosure include skip connections and convolutional layers (e.g., between transformer block stages) that combine high-frequency and low-frequency features from different transformer stages and unify attention to micro-structures, meso-structures and macro-structures. In some aspects, the skip connections enable information propagation in the transformer network.

Multi-stage attention model for texture synthesis

Certain aspects and features of this disclosure relate to rendering images using target-augmented material maps. In one example, a graphics imaging application is loaded with a scene and an input material map, as well as a file for a target image. A stored, material generation prior is accessed by the graphics imaging application. This prior, as an example, is based on a pre-trained, generative adversarial network (GAN). An input material appearance from the input material map is encoded to produce a projected latent vector. The value for the projected latent vector is optimized to produce the material map that is used to render the scene, producing a material map augmented by a realistic target material appearance.

Target-augmented material maps

Embodiments are disclosed for optimizing a material graph for replicating a material of the target image. Embodiments include receiving a target image and a material graph to be optimized for replicating a material of the target image. Embodiments include identifying a non-differentiable node of the material graph, the non-differentiable node including a set of input parameters. Embodiments include selecting a differentiable proxy from a library of the selected differentiable proxy is trained to replicate an output of the identified non-differentiable node. Embodiments include generating an optimized input parameters for the identified non-differentiable node using the corresponding trained neural network and the target image. Embodiments include replacing the set of input parameters of the non-differentiable node of the material graph with the optimized input parameters. Embodiments include generating an output material by the material graph to represent the target image using the optimized input parameters for the non-differentiable node.

Node graph optimization using differentiable proxies

A method of image processing is described. The method comprises receiving a set of at least three images of an object including at least two linearly-polarized images and at least one color image, wherein the three images have the same view of the object and are acquired under the same illumination condition in which either diffuse polarization or specular polarization dominates in surface reflectance, and wherein a set of Stokes parameters s, sand sis determinable from the at least three images. The method further comprises generating three-dimensional shape and spatially-varying reflectance of the object from the set of at least three images using a deep neural network trained with a plurality of sets of training images, each of the plurality of sets of training images including at least three training images including at least two linearly-polarized training images and at least one color image from which a respective set of Stokes parameters s, sand sis determinable and storing said three-dimensional shape and spatially-varying reflectance generated by the deep neural network.

Growing community of inventors

London, United Kingdom

Valentin Deschaintre