Nvidia Corporation

Leland Stanford Junior University

Toyota Research Institute, Inc.

Or Litany

Sanja Fidler

Adrien David Gaidon

Vitor Guizilini

Rares A Ambrus

Jun Gao

Jie Li

Leonidas John Guibas

Wenzheng Chen

Sameh Khamis

Joey Litalien

Zian Wang

Zan Gojcic

Francis Williams

Clement Tse Tsian Christophe Louis Fuji Tsang

Karsten Julian Kreis

Arash Vahdat

Matan Atzmon

Jiahui Huang

Xiaohui Zeng

Approaches presented herein provide for the reconstruction of implicit multi-dimensional shapes. In one embodiment, oriented point cloud data representative of an object can be obtained using a physical scanning process. The point cloud data can be provided as input to a trained density model that can infer density functions for various points. The points can be mapped to a voxel hierarchy, allowing density functions to be determined for those voxels at the various levels that are associated with at least one point of the input point cloud. Contribution weights can be determined for the various density functions for the sparse voxel hierarchy, and the weighted density functions combined to obtain a density field. The density field can be evaluated to generate a geometric mesh where points having a zero, or near-zero, value are determined to contribute to the surface of the object.

Using machine learning for surface reconstruction in synthetic content generation systems and applications

Approaches presented herein provide for the unconditional generation of novel three dimensional (3D) object shape representations, such as point clouds or meshes. In at least one embodiment, a first denoising diffusion model (DDM) can be trained to synthesize a 1D shape latent from Gaussian noise, and a second DDM can be trained to generate a set of latent points conditioned on this 1D shape latent. The shape latent and set of latent points can be provided to a decoder to generate a 3D point cloud representative of a random object from among the object classes on which the models were trained. A surface reconstruction process may be used to generate a surface mesh from this generated point cloud. Such an approach can scale to complex and/or multimodal distributions, and can be highly flexible as it can be adapted to various tasks such as multimodal voxel- or text-guided synthesis.

Synthesizing three-dimensional shapes using latent diffusion models in content generation systems and applications

A method for 3D object detection is described. The method includes concurrently training a monocular depth network and a 3D object detection network. The method also includes predicting, using a trained monocular depth network, a monocular depth map of a monocular image of a video stream. The method further includes inferring a 3D point cloud of a 3D object within the monocular image according to the predicted monocular depth map. The method also includes predicting 3D bounding boxes from a selection of 3D points from the 3D point cloud of the 3D object based on a selection regression loss.

Monocular object detection via end-to-end differentiable pipeline

In various examples, information may be received for a 3D model, such as 3D geometry information, lighting information, and material information. A machine learning model may be trained to disentangle the 3D geometry information, the lighting information, and/or material information from input data to provide the information, which may be used to project geometry of the 3D model onto an image plane to generate a mapping between pixels and portions of the 3D model. Rasterization may then use the mapping to determine which pixels are covered and in what manner, by the geometry. The mapping may also be used to compute radiance for points corresponding to the one or more 3D models using light transport simulation. Disclosed approaches may be used in various applications, such as image editing, 3D model editing, synthetic data generation, and/or data set augmentation.

Hybrid differentiable rendering for light transport simulation systems and applications

A method for 3D object detection is described. The method includes predicting, using a trained monocular depth network, an estimated monocular input depth map of a monocular image of a video stream and an estimated depth uncertainty map associated with the estimated monocular input depth map. The method also includes feeding back a depth uncertainty regression loss associated with the estimated monocular input depth map during training of the trained monocular depth network to update the estimated monocular input depth map. The method further includes detecting 3D objects from a 3D point cloud computed from the estimated monocular input depth map based on seed positions selected from the 3D point cloud and the estimated depth uncertainty map. The method also includes selecting 3D bounding boxes of the 3D objects detected from the 3D point cloud based on the seed positions and an aggregated depth uncertainty.

Systems and methods for uncertainty aware monocular 3D object detection

Growing community of inventors

Sunnyvale, CA, United States of America

Or Litany