Google Inc.

University of California

Christian Haene

Sofien Bouaziz

Shahram Izadi

Cem Keskin

Yinda Zhang

Sameh Khamis

Marc Stewart Levoy

Onur Gonen Guleryuz

Neal Wadhwa

Adarsh Prakash Murthy Kowdle

Rahul Garg

Saurabh Singh

Sean Ryan Francesco Fanello

Vladimir Tankovich

Rohit Kumar Pandey

Yael Pritch Knaan

Sean Ryan Fanello

Chloe LeGendre

Andrea Tagliasacchi

Esther L Yuh

Sergio Orts Escolano

Pratik Mukherjee

Hossam Isack

Jitendra Malik

Peter Barnum

Abhi Meka

Phillip Andrew Chou

Wei-cheng Kuo

Ryan Overbeck

Thabo Beeler

Example embodiments relate to techniques for volumetric performance capture with neural rendering. A technique may involve initially obtaining images that depict a subject from multiple viewpoints and under various lighting conditions using a light stage and depth data corresponding to the subject using infrared cameras. A neural network may extract features of the subject from the images based on the depth data and map the features into a texture space (e.g., the UV texture space). A neural renderer can be used to generate an output image depicting the subject from a target view such that illumination of the subject in the output image aligns with the target view. The neural render may resample the features of the subject from the texture space to an image space to generate the output image.

Volumetric performance capture with neural rendering

Systems and methods are directed to encoding and/or decoding of the textures/geometry of a three-dimensional volumetric representation. An encoding computing system can obtain voxel blocks from a three-dimensional volumetric representation of an object. The encoding computing system can encode voxel blocks with a machine-learned voxel encoding model to obtain encoded voxel blocks. The encoding computing system can decode the encoded voxel blocks with a machine-learned voxel decoding model to obtain reconstructed voxel blocks. The encoding computing system can generate a reconstructed mesh representation of the object based at least in part on the one or more reconstructed voxel blocks. The encoding computing system can encode textures associated with the voxel blocks according to an encoding scheme and based at least in part on the reconstructed mesh representation of the object to obtain encoded textures.

Systems and methods for compression of three-dimensional volumetric representations

Example implementations relate to joint depth prediction from dual cameras and dual pixels. An example method may involve obtaining a first set of depth information representing a scene from a first source and a second set of depth information representing the scene from a second source. The method may further involve determining, using a neural network, a joint depth map that conveys respective depths for elements in the scene. The neural network may determine the joint depth map based on a combination of the first set of depth information and the second set of depth information. In addition, the method may involve modifying an image representing the scene based on the joint depth map. For example, background portions of the image may be partially blurred based on the joint depth map.

Joint depth prediction from dual-cameras and dual-pixels

A computer-implemented method can include a training phase and a hemorrhage detection phase. The training phase can include: receiving a first plurality of frames from at least one original computed tomography (CT) scan of a target subject, wherein each frame may or may not include a visual indication of a hemorrhage, and further wherein each frame including a visual indication of a hemorrhage has at least one label associated therewith; and using a fully convolutional neural network (FCN) to train a model by determining, for each of the first plurality of frames, whether at least one sub-portion of the frame includes a visual indication of a hemorrhage and classifying the sub-portion of the frame based on the determining. The hemorrhage detection phase can include: receiving a second plurality of frames from a CT scan of a target subject, wherein each frame may or may not include a visual indication of a hemorrhage; and determining, for each of the second plurality of frames, whether a plurality of sub-portions of the frame includes a visual indication of a hemorrhage based at least in part on the trained model.

Expert-level detection of acute intracranial hemorrhage on head CT scans

According to an aspect, a real-time active stereo system includes a capture system configured to capture stereo data, where the stereo data includes a first input image and a second input image, and a depth sensing computing system configured to predict a depth map. The depth sensing computing system includes a feature extractor configured to extract features from the first and second images at a plurality of resolutions, an initialization engine configured to generate a plurality of depth estimations, where each of the plurality of depth estimations corresponds to a different resolution, and a propagation engine configured to iteratively refine the plurality of depth estimations based on image warping and spatial propagation.

Real-time stereo matching using a hierarchical iterative refinement network

According to an aspect, a method for pose estimation using a convolutional neural network includes extracting features from an image, downsampling the features to a lower resolution, arranging the features into sets of features, where each set of features corresponds to a separate keypoint of a pose of a subject, updating, by at least one convolutional block, each set of features based on features of one or more neighboring keypoints using a kinematic structure, and predicting the pose of the subject using the updated sets of features.

Human pose estimation using neural networks and kinematic structure

Growing community of inventors

Berkeley, CA, United States of America

Christian Haene