Google Inc.

University of Southern California

Xueming Yu

Paul E Debevec

Graham Fyffe

Shahram Izadi

Adarsh Prakash Murthy Kowdle

Jonathan James Taylor

Sean Ryan Francesco Fanello

Abhijeet Ghosh

Christoph Rhemann

Rohit Kumar Pandey

Jessica Lynn Busch

Jason Angelo Dourgarian

Geoffrey Douglas Harvey

Peter Christopher Lincoln

Mark T Bolas

Sofien Bouaziz

Philip L Davidson

Danhang Tang

Julien Pascal Christophe Valentin

Andrea Tagliasacchi

Mingsong Dou

Emily B Cooper

Jay Busch

Sergio Orts Escolano

Matthew Whalen

Pieter Peers

Christian Häne

Abhimitra Meka

Kaiwen Guo

Peter Joseph Denny

A lighting stage includes a plurality of lights that project alternating spherical color gradient illumination patterns onto an object or human performer at a predetermined frequency. The lighting stage also includes a plurality of cameras that capture images of an object or human performer corresponding to the alternating spherical color gradient illumination patterns. The lighting stage also includes a plurality of depth sensors that capture depth maps of the object or human performer at the predetermined frequency. The lighting stage also includes (or is associated with) one or more processors that implement a machine learning algorithm to produce a three-dimensional (3D) model of the object or human performer. The 3D model includes relighting parameters used to relight the 3D model under different lighting conditions.

Volumetric performance capture with relighting

Methods, systems, and media for relighting images using predicted deep reflectance fields are provided. In some embodiments, the method comprises: identifying a group of training samples, wherein each training sample includes (i) a group of one-light-at-a-time (OLAT) images that have each been captured when one light of a plurality of lights arranged on a lighting structure has been activated, (ii) a group of spherical color gradient images that have each been captured when the plurality of lights arranged on the lighting structure have been activated to each emit a particular color, and (iii) a lighting direction, wherein each image in the group of OLAT images and each of the spherical color gradient images are an image of a subject, and wherein the lighting direction indicates a relative orientation of a light to the subject; training a convolutional neural network using the group of training samples, wherein training the convolutional neural network comprises: for each training iteration in a series of training iterations and for each training sample in the group of training samples: generating an output predicted image, wherein the output predicted image is a representation of the subject associated with the training sample with lighting from the lighting direction associated with the training sample; identifying a ground-truth OLAT image included in the group of OLAT images for the training sample that corresponds to the lighting direction for the training sample; calculating a loss that indicates a perceptual difference between the output predicted image and the identified ground-truth OLAT image; and updating parameters of the convolutional neural network based on the calculated loss; identifying a test sample that includes a second group of spherical color gradient images and a second lighting direction; and generating a relit image of the subject included in each of the second group of spherical color gradient images with lighting from the second lighting direction using the trained convolutional neural network.

Methods, systems, and media for relighting images using predicted deep reflectance fields

An apparatus to measure surface orientation maps of an object may include a light source that is configured to illuminate the object with a controllable field of illumination. One or more cameras may be configured to capture at least one image of the object. A processor may be configured to process the image(s) to extract the reflectance properties of the object including an albedo, a reflection vector, a roughness, and/or anisotropy parameters of a specular reflectance lobe associated with the object. The controllable field of illumination may include limited-order Spherical Harmonics (SH) and Fourier Series (FS) illumination patterns with substantially similar polarization. The SH and FS illumination patterns are used with different light sources.

Specular object scanner for measuring reflectance properties of objects

A controllable lighting system may include a plurality of light source groups, a group controller for each light source group, a master controller, and a network communication system. Each group controller may be configured to control the light sources in its light source group based on a group control command. The master controller may be configured to receive a master control command relating to the light sources and to issue a group control command to each of the group controllers that collectively effectuate compliance with the master control command. The network communication system may be configured to communicate the group control commands from the master controller to the group controllers.

Illumination sphere with intelligent LED lighting units in scalable daisy chain with interchangeable filters

Growing community of inventors

Arcadia, CA, United States of America

Xueming Yu