Nvidia Corporation

Other

University of Toronto

Wenzheng Chen

Sanja Fidler

Jun Gao

Huan Ling

Zian Wang

David Jesus Acuna Marrero

Amlan Kar

Or Litany

Joey Litalien

Sameh Khamis

Clement Tse Tsian Christophe Louis Fuji Tsang

Jan Kautz

Carl Jacob Munkberg

Jon Niklas Theodor Hasselgren

Kiriakos Neoklis Kutulakos

Antonio Torralba Barriuso

Tianchang Shen

Alex John Bauld Evans

David Acuna Marrero

Yuxuan Zhang

Seyed Parsa Mirdehghan

Thomas Müller-Höhne

Systems and methods generate a hybrid lighting model for rendering objects within an image. The hybrid lighting model includes lighting effects attributed to a first source, such as the sun, and to a second source, such as spatially-varying effects of objects within the image. The hybrid lighting model may be generated for an input image and then one or more virtual objects may be rendered to appear as if part of the input image, where the hybrid lighting model is used to apply one or more lighting effects to the one or more virtual objects.

Differentiable object insertion using hybrid lighting volumes for synthetic data generation

In various examples, information may be received for a 3D model, such as 3D geometry information, lighting information, and material information. A machine learning model may be trained to disentangle the 3D geometry information, the lighting information, and/or material information from input data to provide the information, which may be used to project geometry of the 3D model onto an image plane to generate a mapping between pixels and portions of the 3D model. Rasterization may then use the mapping to determine which pixels are covered and in what manner, by the geometry. The mapping may also be used to compute radiance for points corresponding to the one or more 3D models using light transport simulation. Disclosed approaches may be used in various applications, such as image editing, 3D model editing, synthetic data generation, and/or data set augmentation.

Hybrid differentiable rendering for light transport simulation systems and applications

A technique is described for extracting or constructing a three-dimensional (3D) model from multiple two-dimensional (2D) images. In an embodiment, a foreground segmentation mask or depth field may be provided as an additional supervision input with each 2D image. In an embodiment, the foreground segmentation mask or depth field is automatically generated for each 2D image. The constructed 3D model comprises a triangular mesh topology, materials, and environment lighting. The constructed 3D model is represented in a format that can be directly edited and/or rendered by conventional application programs, such as digital content creation (DCC) tools. For example, the constructed 3D model may be represented as a triangular surface mesh (with arbitrary topology), a set of 2D textures representing spatially-varying material parameters, and an environment map. Furthermore, the constructed 3D model may be included in 3D scenes and interacts realistically with other objects.

Extracting triangular 3-D models, materials, and lighting from images

The present invention relates generally to object annotation, specifically to polygonal annotations of objects. Described are methods of annotating an object including steps of receiving an image depicting an object, generating a set of image features using a CNN encoder implemented on one or more computers, and producing a polygon object annotation via a recurrent decoder or a Graph Neural Network. The recurrent decoder may include a recurrent neural network, a graph neural network or a gated graph neural network. A system for annotating an object and a method of training an object annotation system are also described.

Systems and methods for polygon object annotation and a method of training an object annotation system

Approaches are presented for training an inverse graphics network. An image synthesis network can generate training data for an inverse graphics network. In turn, the inverse graphics network can teach the synthesis network about the physical three-dimensional (3D) controls. Such an approach can provide for accurate 3D reconstruction of objects from 2D images using the trained inverse graphics network, while requiring little annotation of the provided training data. Such an approach can extract and disentangle 3D knowledge learned by generative models by utilizing differentiable renderers, enabling a disentangled generative model to function as a controllable 3D 'neural renderer,' complementing traditional graphics renderers.

Neural rendering for inverse graphics generation

There is provided a system and method for optimizing depth imaging. The method including: illuminating one or more scenes with illumination patterns; capturing one or more images of each of the scenes; reconstructing the scenes; estimating the reconstruction error and a gradient of the reconstruction error; iteratively performing until the reconstruction error reaches a predetermined error condition: determining a current set of control vectors and current set of reconstruction parameters; illuminating the one or more scenes with the illumination patterns governed by the current set of control vectors; capturing one or more images of each of the scenes while the scene is being illuminated with at least one of the illumination patterns; reconstructing the scenes from the one or more captured images using the current reconstruction parameters; and estimating an updated reconstruction error and gradient; and outputting at least one of control vectors and reconstruction parameters.

Method and system for optimizing depth imaging

Systems and methods for polygon object annotation and a method of training and object annotation system

Growing community of inventors

Toronto, Canada

Wenzheng Chen