Google Inc.

Yinda Zhang

Shahram Izadi

Sofien Bouaziz

Sean Ryan Francesco Fanello

Christian Haene

Christoph Rhemann

Vladimir Tankovich

Sean Ryan Fanello

Sameh Khamis

Ruofei Du

Rohit Kumar Pandey

Cem Keskin

Michael John Schoenberg

Danhang Tang

Mingsong Dou

Julien Valentin

Thomas Allen Funkhouser

Adarsh Kowdle

Marc Stewart Levoy

Onur Gonen Guleryuz

Neal Wadhwa

Adarsh Prakash Murthy Kowdle

Rahul Garg

Saurabh Singh

Jonathan James Taylor

Philip L Davidson

Yael Pritch Knaan

Chloe LeGendre

Andrea Tagliasacchi

Sergio Orts Escolano

Deqing Sun

Peter Barnum

Abhi Meka

Feitong Tan

Phillip Andrew Chou

Kaiwen Guo

Ryan Overbeck

Thabo Beeler

Example embodiments relate to techniques for volumetric performance capture with neural rendering. A technique may involve initially obtaining images that depict a subject from multiple viewpoints and under various lighting conditions using a light stage and depth data corresponding to the subject using infrared cameras. A neural network may extract features of the subject from the images based on the depth data and map the features into a texture space (e.g., the UV texture space). A neural renderer can be used to generate an output image depicting the subject from a target view such that illumination of the subject in the output image aligns with the target view. The neural render may resample the features of the subject from the texture space to an image space to generate the output image.

Volumetric performance capture with neural rendering

Systems and methods are directed to encoding and/or decoding of the textures/geometry of a three-dimensional volumetric representation. An encoding computing system can obtain voxel blocks from a three-dimensional volumetric representation of an object. The encoding computing system can encode voxel blocks with a machine-learned voxel encoding model to obtain encoded voxel blocks. The encoding computing system can decode the encoded voxel blocks with a machine-learned voxel decoding model to obtain reconstructed voxel blocks. The encoding computing system can generate a reconstructed mesh representation of the object based at least in part on the one or more reconstructed voxel blocks. The encoding computing system can encode textures associated with the voxel blocks according to an encoding scheme and based at least in part on the reconstructed mesh representation of the object to obtain encoded textures.

Systems and methods for compression of three-dimensional volumetric representations

Example implementations relate to joint depth prediction from dual cameras and dual pixels. An example method may involve obtaining a first set of depth information representing a scene from a first source and a second set of depth information representing the scene from a second source. The method may further involve determining, using a neural network, a joint depth map that conveys respective depths for elements in the scene. The neural network may determine the joint depth map based on a combination of the first set of depth information and the second set of depth information. In addition, the method may involve modifying an image representing the scene based on the joint depth map. For example, background portions of the image may be partially blurred based on the joint depth map.

Joint depth prediction from dual-cameras and dual-pixels

An electronic device estimates a depth map of an environment based on matching reduced-resolution stereo depth images captured by depth cameras to generate a coarse disparity (depth) map. The electronic device downsamples depth images captured by the depth cameras and matches sections of the reduced-resolution images to each other to generate a coarse depth map. The electronic device upsamples the coarse depth map to a higher resolution and refines the upsampled depth map to generate a high-resolution depth map to support location-based functionality.

Active stereo depth prediction based on coarse matching

A method can include selecting, from at least a first avatar and a second avatar based on at least one attribute of a calendar event associated with a user, a session avatar, the first avatar being based on a first set of images of a user wearing a first outfit and the second avatar being based on a second set of images of the user wearing a second outfit, and presenting the session avatar during a videoconference, the presentation of the session avatar changing based on audio input received from the user during the videoconference.

Selecting avatar for videoconference

Systems and methods for training models to predict dense correspondences across images such as human images. A model may be trained using synthetic training data created from one or more 3D computer models of a subject. In addition, one or more geodesic distances derived from the surfaces of one or more of the 3D models may be used to generate one or more loss values, which may in turn be used in modifying the model's parameters during training.

Systems and methods for training models to predict dense correspondences in images using geodesic distances

According to an aspect, a real-time active stereo system includes a capture system configured to capture stereo data, where the stereo data includes a first input image and a second input image, and a depth sensing computing system configured to predict a depth map. The depth sensing computing system includes a feature extractor configured to extract features from the first and second images at a plurality of resolutions, an initialization engine configured to generate a plurality of depth estimations, where each of the plurality of depth estimations corresponds to a different resolution, and a propagation engine configured to iteratively refine the plurality of depth estimations based on image warping and spatial propagation.

Real-time stereo matching using a hierarchical iterative refinement network

Growing community of inventors

Mountain View, CA, United States of America

Yinda Zhang