Nvidia Corporation

Zan Gojcic

Sanja Fidler

Or Litany

Francis Williams

Ignacio Llamas

Karsten Julian Kreis

Arash Vahdat

Zian Wang

Gavriel State

Matan Atzmon

Jiahui Huang

Nicolas Moenne-Loccoz

Xiaohui Zeng

Approaches presented herein provide for the reconstruction of implicit multi-dimensional shapes. In one embodiment, oriented point cloud data representative of an object can be obtained using a physical scanning process. The point cloud data can be provided as input to a trained density model that can infer density functions for various points. The points can be mapped to a voxel hierarchy, allowing density functions to be determined for those voxels at the various levels that are associated with at least one point of the input point cloud. Contribution weights can be determined for the various density functions for the sparse voxel hierarchy, and the weighted density functions combined to obtain a density field. The density field can be evaluated to generate a geometric mesh where points having a zero, or near-zero, value are determined to contribute to the surface of the object.

Using machine learning for surface reconstruction in synthetic content generation systems and applications

Approaches presented herein provide for the unconditional generation of novel three dimensional (3D) object shape representations, such as point clouds or meshes. In at least one embodiment, a first denoising diffusion model (DDM) can be trained to synthesize a 1D shape latent from Gaussian noise, and a second DDM can be trained to generate a set of latent points conditioned on this 1D shape latent. The shape latent and set of latent points can be provided to a decoder to generate a 3D point cloud representative of a random object from among the object classes on which the models were trained. A surface reconstruction process may be used to generate a surface mesh from this generated point cloud. Such an approach can scale to complex and/or multimodal distributions, and can be highly flexible as it can be adapted to various tasks such as multimodal voxel- or text-guided synthesis.

Synthesizing three-dimensional shapes using latent diffusion models in content generation systems and applications

Approaches presented herein provide for the generation of visual content, including different types of content representations from different sources, rendered to include consistent scene illumination for the various representations. A first render pass can produce a first image including only proxies of implicit representations (e.g., NeRF objects) under scene illumination. A second render pass can produce a second image that includes a representation of the explicit scene objects, as well as the proxies of the implicit representations, under the scene illumination, which produces secondary lighting effects. The first and second images are compared to determine irradiance ratio data for the various pixel locations. A third render pass can produce a third image that includes the implicit representations, which can have relighting performed according to the irradiance ratio data to include the secondary lighting effects. The implicit and explicit objects can then be composited to produce an image with consistent scene illumination.

Rendering and composition of neural 3D objects with non-neural assets in content generation systems and applications

Growing community of inventors

Zurich, Switzerland

Zan Gojcic