Massachusetts Institute of Technology

Google Inc.

Mitsubishi Electric Research Laboratories, Inc.

Quanta Computer Incorporated

Adobe, Inc.

Mitsubishi Electric Information Technology Center America, Inc.

Mitsubishi Electric Information Technology Center America, Inc (ita)

Other

Mitsubishi Electric Research Labs.

Shell Oil Comapny

Toyota Research Institute, Inc.

University of Toronto

Qatar Foundation for Education, Science and Community Development

William T Freeman

Frederic Durand

Michael Rubinstein

Neal Wadhwa

Forrester H Cole

Ce Liu

Tali Dekel

John V Guttag

Myers Abraham Davis

Eugene Inghaw Shih

Hao-yu Wu

Wojciech Matusik

Richard Tucker

Shmuel Avidan

Rahul Sukthankar

Dilip Krishnan

David Benjamin Belanger

Pitchaya Sitthi-Amorn

Oral Buyukozturk

Noah Snavely

Zhengqi Li

Justin Gejune Chen

Cristian Sminchisescu

Andrei Zanfir

Taeg Sang Cho

Eduard Gabriel Bazavan

Egon C Pasztor

Tianfan Xue

William Yerazunis

Jonathan S Yedidia

Kevin William Wilson

Samuel William Hasinoff

Fredo Durand

Avinatan Hassidim

Inbar Mosseri

Szu-Po Wang

Petr Kellnhofer

David Salesin

Ariel Ephrat

Oran Lang

Mihai Zanfir

Sourabh A Niyogi

Walter A Hiatt

Piotr K Didyk

YiChang Shih

Guy Rosman

Daniela Rus

Sylvain Philippe Paris

Edward H Adelson

Joseph W Marks

Gregory W Wornell

Dominik Philemon Kaeser

Baback Moghaddam

Sertac Karaman

Mohamed Hefeeda

Alexei A Efros

Igor Gilitschenski

Thouis R Jones

Michael Krainin

Huiwen Chang

Theodericus Johannes Henricus Smit

Antonio Torralba

Alexander Amini

John William Fisher, Iii

Brian Curless

Abhishek Kar

Piotr Krzysztof Didyk

Mohamed Abdelaziz A Mohamed Elgharib

Michael E Leventon

Felix Maximilian Naser

Kyle Sargent

Paul E Green

Paul Beardsley

Donglai Wei

Erika Lu

Zhoutong Zhang

Hongyi Xu

Varun Jampani

Joshua B Tennenbaum

Sergio Daniel Kapusta

Reza Mohammadi Ghazi

James Long

Zoran Dzunic

Anat Levin

Peter Sand

Hossein Mobahi

Robert D W Fergus

Barun Singh

Aaron Phillip Hertzmann

Sam T Roweis

Yair Weiss

Craig D Weissman

David Henry Salesin

A method includes determining, based on a first image, a first depth of a first pixel and, based on a second image, a second depth of a second pixel that corresponds to the first pixel. The method also includes determining a first 3D point based on the first depth and a second 3D point based on the second depth, and determining a scene flow between the first and second images. The method additionally includes determining an induced pixel position based on a post-flow 3D point representing the first 3D point displaced according to the scene flow, determining a flow loss value based on the induced pixel position and a position of the second pixel and a depth loss value based on the post-flow 3D point and the second 3D point, and adjusting the depth model or the scene flow model based on the flow and depth loss values.

Model for determining consistent depth of moving objects in video

A method includes determining, based on an image having an initial viewpoint, a depth image, and determining a foreground visibility map including visibility values that are inversely proportional to a depth gradient of the depth image. The method also includes determining, based on the depth image, a background disocclusion mask indicating a likelihood that pixel of the image will be disoccluded by a viewpoint adjustment. The method additionally includes generating, based on the image, the depth image, and the background disocclusion mask, an inpainted image and an inpainted depth image. The method further includes generating, based on the depth image and the inpainted depth image, respectively, a first three-dimensional (3D) representation of the image and a second 3D representation of the inpainted image, and generating a modified image having an adjusted viewpoint by combining the first and second 3D representation based on the foreground visibility map.

Single image 3D photography with soft-layering and depth-aware inpainting

The present disclosure provides systems and methods that perform face reconstruction based on an image of a face. In particular, one example system of the present disclosure combines a machine-learned image recognition model with a face modeler that uses a morphable model of a human's facial appearance. The image recognition model can be a deep learning model that generates an embedding in response to receipt of an image (e.g., an uncontrolled image of a face). The example system can further include a small, lightweight, translation model structurally positioned between the image recognition model and the face modeler. The translation model can be a machine-learned model that is trained to receive the embedding generated by the image recognition model and, in response, output a plurality of facial modeling parameter values usable by the face modeler to generate a model of the face.

Face reconstruction from a learned embedding

The present disclosure provides a statistical, articulated 3D human shape modeling pipeline within a fully trainable, modular, deep learning framework. In particular, aspects of the present disclosure are directed to a machine-learned 3D human shape model with at least facial and body shape components that are jointly trained end-to-end on a set of training data. Joint training of the model components (e.g., including both facial, hands, and rest of body components) enables improved consistency of synthesis between the generated face and body shapes.

Generative nonlinear human shape models

A computer-implemented method for decomposing videos into multiple layers () that can be re-combined with modified relative timings includes obtaining video data including a plurality of image frames () depicting one or more objects. For each of the plurality of frames, the computer-implemented method includes generating one or more object maps descriptive of a respective location of at least one object of the one or more objects within the image frame. For each of the plurality of frames, the computer-implemented method includes inputting the image frame and the one or more object maps into a machine-learned layer Tenderer model. () For each of the plurality of frames, the computer-implemented method includes receiving, as output from the machine-learned layer Tenderer model, a background layer illustrative of a background of the video data and one or more object layers respectively associated with one of the one or more object maps. The object layers include image data illustrative of the at least one object and one or more trace effects at least partially attributable to the at least one object such that the one or more object layers and the background layer can be re-combined with modified relative timings.

Re-timing objects in video via layered neural rendering

A method includes obtaining a reference image and a target image each representing an environment containing moving features and static features. The method also includes determining an object mask configured to mask out the moving features and preserves the static features in the target image. The method additionally includes determining, based on motion parallax between the reference image and the target image, a static depth image representing depth values of the static features in the target image. The method further includes generating, by way of a machine learning model, a dynamic depth image representing depth values of both the static features and the moving features in the target image. The model is trained to generate the dynamic depth image by determining depth values of at least the moving features based on the target image, the object mask, and the static depth image.

Depth determination for images captured with a moving camera and representing moving features

The present disclosure is generally directed to reconstructing representations of bodies from images. An example method of the present disclosure includes inputting, into a machine-learned reconstruction model, input data descriptive of an image depicting a body; predicting, using a machine-learned marker prediction component of the reconstruction model, a set of surface marker locations on the body; and outputting, using a machine-learned marker poser component of the reconstruction model, an output representation of the body that corresponds to the set of surface marker locations. In the example method, one or more parameters of the reconstruction model were learned at least in part based on a consistency loss corresponding to a distance between relaxed-constraint representations generated from a prior set of surface marker locations predicted according to the one or more parameters and parametric representations generated from the prior set using kinematic constraints associated with the body.

Systems and methods for reconstructing body shape and pose

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for audio-visual speech separation. A method includes: obtaining, for each frame in a stream of frames from a video in which faces of one or more speakers have been detected, a respective per-frame face embedding of the face of each speaker; processing, for each speaker, the per-frame face embeddings of the face of the speaker to generate visual features for the face of the speaker; obtaining a spectrogram of an audio soundtrack for the video; processing the spectrogram to generate an audio embedding for the audio soundtrack; combining the visual features for the one or more speakers and the audio embedding for the audio soundtrack to generate an audio-visual embedding for the video; determining a respective spectrogram mask for each of the one or more speakers; and determining a respective isolated speech spectrogram for each speaker.

Audio-visual speech separation

Systems and methods are directed to a method for estimation of an object state from image data. The method can include obtaining two-dimensional image data depicting an object. The method can include processing, with an estimation portion of a machine-learned object state estimation model, the two-dimensional image data to obtain an initial estimated state of the object. The method can include, for each of one or more refinement iterations, obtaining a previous loss value associated with a previous estimated state for the object, processing the previous loss value to obtain a current estimated state of the object, and evaluating a loss function to determine a loss value associated with the current estimated state of the object. The method can include providing a final estimated state for the object.

Systems and methods for refined object estimation from image data

The present disclosure provides systems and methods to detect occluded objects using shadow information to anticipate moving obstacles that are occluded behind a corner or other obstacle. The system may perform a dynamic threshold analysis on enhanced images allowing the detection of even weakly visible shadows. The system may classify an image sequence as either 'dynamic' or 'static', enabling an autonomous vehicle, or other moving platform, to react and respond to a moving, yet occluded object by slowing down or stopping.

Systems and methods of detecting moving obstacles

The present disclosure relates to systems and methods for image capture. Namely, an image capture system may include a camera configured to capture images of a field of view, a display, and a controller. An initial image of the field of view from an initial camera pose may be captured. An obstruction may be determined to be observable in the field of view. Based on the obstruction, at least one desired camera pose may be determined. The at least one desired camera pose includes at least one desired position of the camera. A capture interface may be displayed, which may include instructions for moving the camera to the at least one desired camera pose. At least one further image of the field of view from the at least one desired camera pose may be captured. Captured images may be processed to remove the obstruction from a background image.

Taking photos through visual obstructions

Growing community of inventors

Acton, MA, United States of America

William T Freeman