Massachusetts Institute of Technology

Everyscape, Inc.

Mitsubishi Electric Research Laboratories, Inc.

Toyota Research Institute, Inc.

Other

Leland Stanford Junior University

Mok3, Inc.

Fredo Durand

Byong Mok Oh

Wojciech Matusik

William T Freeman

Hanspeter Pfister

Adrien David Gaidon

Vitor Guizilini

Rares A Ambrus

Guy Rosman

Daniela Rus

Gregory W Wornell

Sergey Zakharov

Dennis Park

Sertac Karaman

Matthias Zwicker

Igor Gilitschenski

Max G Chen

Morgan S McGuire

Antonio Torralba

Alexander Amini

Taeg Sang Cho

John F Hughes

Vincent Sitzmann

Felix Maximilian Naser

Anat Levin

Peter Sand

Wadim Kehl

Joshua Tenenbaum

Jiajun Wu

Systems and methods described herein relate to reconstructing a scene in three dimensions from a two-dimensional image. One embodiment processes an image using a detection transformer to detect an object in the scene and to generate a NOCS map of the object and a background depth map; uses MLPs to relate the object to a differentiable database of object priors (PriorDB); recovers, from the NOCS map, a partial 3D object shape; estimates an initial object pose; fits a PriorDB object prior to align in geometry and appearance with the partial 3D shape to produce a complete shape and refines the initial pose estimate; generates an editable and re-renderable 3D scene reconstruction based, at least in part, on the complete shape, the refined pose estimate, and the depth map; and controls the operation of a robot based, at least in part, on the editable and re-renderable 3D scene reconstruction.

Systems and methods for reconstructing a scene in three dimensions from a two-dimensional image

The present disclosure provides systems and methods to detect occluded objects using shadow information to anticipate moving obstacles that are occluded behind a corner or other obstacle. The system may perform a dynamic threshold analysis on enhanced images allowing the detection of even weakly visible shadows. The system may classify an image sequence as either 'dynamic' or 'static', enabling an autonomous vehicle, or other moving platform, to react and respond to a moving, yet occluded object by slowing down or stopping.

Systems and methods of detecting moving obstacles

Object motion during camera exposure often leads to noticeable blurring artifacts. Proper elimination of this blur is challenging because the blur kernel is unknown, varies over the image as a function of object velocity, and destroys high frequencies. In the case of motions along a 1D direction (e.g. horizontal), applicants show that these challenges can be addressed using a camera that moves during the exposure. Through the analysis of motion blur as space-time integration, applicants show that a parabolic integration (corresponding to constant sensor acceleration) leads to motion blur that is not only invariant to object velocity, but preserves image frequency content nearly optimally. That is, static objects are degraded relative to their image from a static camera, but all moving objects within a given range of motions reconstruct well. A single deconvolution kernel can be used to remove blur and create sharp images of scenes with objects moving at different speeds, without requiring any segmentation and without knowledge of the object speeds.

Method and apparatus for motion invariant imaging

The invention provides tools and techniques for clone brushing pixels in an image while accounting for inconsistencies in apparent depth and orientation within the image. The techniques do not require any depth information to be present in the image, and the data structure of the image is preserved. The techniques allow for color compensation between source and destination regions. A snapping technique is also provided to facilitate increased accuracy in selecting source and destination positions.

Structure-preserving clone brush

A method and system acquire and display light fields. A continuous light field is reconstructed from input samples of an input light field of a 3D scene acquired by cameras according to an acquisition parameterization. The continuous light is reparameterized according to a display parameterization and then prefiltering and sampled to produce output samples having the display parametrization. The output samples are displayed as an output light field using a 3D display device.

Method and system for acquiring and displaying 3D light fields

A method compresses a set of correlated signals by first converting each signal to a sequence of integers, which are further organized as a set of bit-planes. An inverse accumulator is applied to each bit-plane to produce a bit-plane of shifted bits, which are permuted according to a predetermined permutation to produce bit-planes of permuted bits. Each bit-plane of permuted bits is partitioned into a set of blocks of bits. Syndrome bits are generated for each block of bits according to a rate-adaptive base code. Subsequently, the syndrome bits are decompressed in a decoder to recover the original correlated signals.

System and method for image matting

The invention provides a variety of tools and techniques for adding depth information to photographic images, and for editing and manipulating images that include depth information. The tools for working with such images include tools for 'painting' in a depth channel, for using geometric primitives and other three-dimensional shapes to define depth in a two-dimensional image, and tools for 'clone brushing' portions of an image with depth information while taking the depth information and lighting into account when copying from one portion of the image to another. The tools also include relighting tools that separate illumination information from texture information.

Image-based modeling and photo editing

Growing community of inventors

Boston, MA, United States of America

Fredo Durand