Google Inc.

Sean Ryan Fanello

Shahram Izadi

Vladimir Tankovich

Christoph Rhemann

Adarsh Prakash Murthy Kowdle

Philip L Davidson

Sameh Khamis

Julien Pascal Christophe Valentin

Yinda Zhang

Michael John Schoenberg

Mingsong Dou

Julien Valentin

Thomas Allen Funkhouser

Adarsh Kowdle

Marc Stewart Levoy

Jonathan Tilton Barron

Neal Wadhwa

Rahul Garg

Jonathan James Taylor

Yael Pritch Knaan

David Kim

Yun-Ta Tsai

Cem Keskin

Danhang Tang

Tianfan Xue

Christian Haene

Tiancheng Sun

Sergio Orts Escolano

Xiuming Zhang

Example implementations relate to joint depth prediction from dual cameras and dual pixels. An example method may involve obtaining a first set of depth information representing a scene from a first source and a second set of depth information representing the scene from a second source. The method may further involve determining, using a neural network, a joint depth map that conveys respective depths for elements in the scene. The neural network may determine the joint depth map based on a combination of the first set of depth information and the second set of depth information. In addition, the method may involve modifying an image representing the scene based on the joint depth map. For example, background portions of the image may be partially blurred based on the joint depth map.

Joint depth prediction from dual-cameras and dual-pixels

An electronic device estimates a depth map of an environment based on matching reduced-resolution stereo depth images captured by depth cameras to generate a coarse disparity (depth) map. The electronic device downsamples depth images captured by the depth cameras and matches sections of the reduced-resolution images to each other to generate a coarse depth map. The electronic device upsamples the coarse depth map to a higher resolution and refines the upsampled depth map to generate a high-resolution depth map to support location-based functionality.

Active stereo depth prediction based on coarse matching

Examples relate to implementations of a neural light transport. A computing system may obtain data indicative of a plurality of UV texture maps and a geometry of an object. Each UV texture map depicts the object from a perspective of a plurality of perspectives. The computing system may train a neural network to learn a light transport function using the data. The light transport function may be a continuous function that specifies how light interacts with the object when the object is viewed from the plurality of perspectives. The computing system may generate an output UV texture map that depicts the object from a synthesized perspective based on an application of the light transport function by the trained neural network.

Neural light transport

Values of pixels in an image are mapped to a binary space using a first function that preserves characteristics of values of the pixels. Labels are iteratively assigned to the pixels in the image in parallel based on a second function. The label assigned to each pixel is determined based on values of a set of nearest-neighbor pixels. The first function is trained to map values of pixels in a set of training images to the binary space and the second function is trained to assign labels to the pixels in the set of training images. Considering only the nearest neighbors in the inference scheme results in a computational complexity that is independent of the size of the solution space and produces sufficient approximations of the true distribution when the solution for each pixel is most likely found in a small subset of the set of potential solutions.

Fully parallel, low complexity approach to solving computer vision problems

An electronic device estimates a pose of one or more subjects in an environment based on estimating a correspondence between a data volume containing a data mesh based on a current frame captured by a depth camera and a reference volume containing a plurality of fused prior data frames based on spectral embedding and performing bidirectional non-rigid matching between the reference volume and the current data frame to refine the correspondence so as to support location-based functionality. The electronic device predicts correspondences between the data volume and the reference volume based on spectral embedding. The correspondences provide constraints that accelerate the convergence between the data volume and the reference volume. By tracking changes between the current data mesh frame and the reference volume, the electronic device avoids tracking failures that can occur when relying solely on a previous data mesh frame.

Non-rigid alignment for volumetric performance capture

An electronic device estimates a depth map of an environment based on stereo depth images captured by depth cameras having exposure times that are offset from each other in conjunction with illuminators pulsing illumination patterns into the environment. A processor of the electronic device matches small sections of the depth images from the cameras to each other and to corresponding patches of immediately preceding depth images (e.g., a spatio-temporal image patch 'cube'). The processor computes a matching cost for each spatio-temporal image patch cube by converting each spatio-temporal image patch into binary codes and defining a cost function between two stereo image patches as the difference between the binary codes. The processor minimizes the matching cost to generate a disparity map, and optimizes the disparity map by rejecting outliers using a decision tree with learned pixel offsets and refining subpixels to generate a depth map of the environment.

System and method for active stereo depth sensing

A first and second image of a scene are captured. Each of a plurality of pixels in the first image is associated with a disparity value. An image patch associated with each of the plurality of pixels of the first image and the second image is mapped into a binary vector. Thus, values of pixels in an image are mapped to a binary space using a function that preserves characteristics of values of the pixels. The difference between the binary vector associated with each of the plurality of pixels of the first image and its corresponding binary vector in the second image designated by the disparity value associated with each of the plurality of pixels of the first image is determined. Based on the determined difference between binary vectors, correspondence between the plurality of pixels of the first image and the second image is established.

Learning-based matching for active stereo systems

Growing community of inventors

Mountain View, CA, United States of America

Sean Ryan Fanello