Google Inc.

Sergio Orts Escolano

Shahram Izadi

Sean Ryan Francesco Fanello

Rohit Kumar Pandey

Christoph Rhemann

Yinda Zhang

Christian Haene

Marc Stewart Levoy

Paul E Debevec

Sofien Bouaziz

Neal Wadhwa

Adarsh Prakash Murthy Kowdle

Rahul Garg

Jonathan James Taylor

Philip L Davidson

Graham Fyffe

Yael Pritch Knaan

Sean Ryan Fanello

Chloe LeGendre

Danhang Tang

Mingsong Dou

Emily B Cooper

Xueming Yu

Jessica Lynn Busch

Jason Angelo Dourgarian

Geoffrey Douglas Harvey

Peter Christopher Lincoln

Peter Barnum

Abhi Meka

Kaiwen Guo

Ryan Overbeck

Thabo Beeler

Example embodiments relate to techniques for volumetric performance capture with neural rendering. A technique may involve initially obtaining images that depict a subject from multiple viewpoints and under various lighting conditions using a light stage and depth data corresponding to the subject using infrared cameras. A neural network may extract features of the subject from the images based on the depth data and map the features into a texture space (e.g., the UV texture space). A neural renderer can be used to generate an output image depicting the subject from a target view such that illumination of the subject in the output image aligns with the target view. The neural render may resample the features of the subject from the texture space to an image space to generate the output image.

Volumetric performance capture with neural rendering

Example implementations relate to joint depth prediction from dual cameras and dual pixels. An example method may involve obtaining a first set of depth information representing a scene from a first source and a second set of depth information representing the scene from a second source. The method may further involve determining, using a neural network, a joint depth map that conveys respective depths for elements in the scene. The neural network may determine the joint depth map based on a combination of the first set of depth information and the second set of depth information. In addition, the method may involve modifying an image representing the scene based on the joint depth map. For example, background portions of the image may be partially blurred based on the joint depth map.

Joint depth prediction from dual-cameras and dual-pixels

A lighting stage includes a plurality of lights that project alternating spherical color gradient illumination patterns onto an object or human performer at a predetermined frequency. The lighting stage also includes a plurality of cameras that capture images of an object or human performer corresponding to the alternating spherical color gradient illumination patterns. The lighting stage also includes a plurality of depth sensors that capture depth maps of the object or human performer at the predetermined frequency. The lighting stage also includes (or is associated with) one or more processors that implement a machine learning algorithm to produce a three-dimensional (3D) model of the object or human performer. The 3D model includes relighting parameters used to relight the 3D model under different lighting conditions.

Volumetric performance capture with relighting

Growing community of inventors

Mountain View, CA, United States of America

Sergio Orts Escolano