Massachusetts Institute of Technology

Toyota Research Institute, Inc.

Toyota Jidosha Kabushiki Kaisha

Leland Stanford Junior University

Vincent Sitzmann

Vitor Guizilini

Rares A Ambrus

Sergey Zakharov

Adrien David Gaidon

Dennis Park

Fredo Durand

Jiading Fang

Wadim Kehl

Joshua Tenenbaum

Jiajun Wu

Systems and methods described herein support enhanced computer vision capabilities which may be applicable to, for example, autonomous vehicle operation. An example method includes generating a latent space and a decoder based on image data that includes multiple images, where each image has a different viewing frame of a scene. The method also includes generating a volumetric embedding that is representative of a novel viewing frame of the scene. The method includes decoding, with the decoder, the latent space using cross-attention with the volumetric embedding, and generating a novel viewing frame of the scene based on an output of the decoder.

Cross-attention decoding for volumetric rendering

Systems and methods described herein relate to reconstructing a scene in three dimensions from a two-dimensional image. One embodiment processes an image using a detection transformer to detect an object in the scene and to generate a NOCS map of the object and a background depth map; uses MLPs to relate the object to a differentiable database of object priors (PriorDB); recovers, from the NOCS map, a partial 3D object shape; estimates an initial object pose; fits a PriorDB object prior to align in geometry and appearance with the partial 3D shape to produce a complete shape and refines the initial pose estimate; generates an editable and re-renderable 3D scene reconstruction based, at least in part, on the complete shape, the refined pose estimate, and the depth map; and controls the operation of a robot based, at least in part, on the editable and re-renderable 3D scene reconstruction.

Systems and methods for reconstructing a scene in three dimensions from a two-dimensional image

Growing community of inventors

Cambridge, MA, United States of America

Vincent Sitzmann