Massachusetts Institute of Technology

Leland Stanford Junior University

Toyota Research Institute, Inc.

Vincent Sitzmann

Adrien David Gaidon

Vitor Guizilini

Rares A Ambrus

Dennis Park

Sergey Zakharov

Fredo Durand

Joshua Tenenbaum

Jiajun Wu

Wadim Kehl

Systems and methods described herein relate to reconstructing a scene in three dimensions from a two-dimensional image. One embodiment processes an image using a detection transformer to detect an object in the scene and to generate a NOCS map of the object and a background depth map; uses MLPs to relate the object to a differentiable database of object priors (PriorDB); recovers, from the NOCS map, a partial 3D object shape; estimates an initial object pose; fits a PriorDB object prior to align in geometry and appearance with the partial 3D shape to produce a complete shape and refines the initial pose estimate; generates an editable and re-renderable 3D scene reconstruction based, at least in part, on the complete shape, the refined pose estimate, and the depth map; and controls the operation of a robot based, at least in part, on the editable and re-renderable 3D scene reconstruction.

Systems and methods for reconstructing a scene in three dimensions from a two-dimensional image

Growing community of inventors

Cambridge, MA, United States of America

Vincent Sitzmann