Lucasfilm Entertainment Company Ltd.

Jeffery Bruce Yost

Kiran Bhat

Michael Koperwas

Ji Hun Yu

Sheila Santos

Steve Sullivan

Cary Phillips

Ronald Fedkiw

Max S-Han Chen

Leandro F Estebecorena

Pablo Helman

Per Karefelt

Stéphane Grabli

Michael Bao

Adam Ferrall-Nunge

A method of transferring a facial expression from a subject to a computer generated character that includes receiving a plate with an image of the subject's facial expression, a three-dimensional parameterized deformable model of the subject's face where different facial expressions of the subject can be obtained by varying values of the model parameters, a model of a camera rig used to capture the plate, and a virtual lighting model that estimates lighting conditions when the image on the plate was captured. The method can solve for the facial expression in the plate by executing a deformation solver to solve for at least some parameters of the deformable model with a differentiable renderer and shape from shading techniques, using, as inputs, the three-dimensional parameterized deformable model, the model of the camera rig and the virtual lighting model over a series of iterations to infer geometry of the facial expression and generate a final facial mesh using the set of parameter values of the deformable model which result in a facial expression that closely matches the expression of the subject in the plate.

On-set facial performance capture and transfer to a three-dimensional computer-generated model

Performance capture systems and techniques are provided for capturing a performance of a subject and reproducing an animated performance that tracks the subject's performance. For example, systems and techniques are provided for determining control values for controlling an animation model to define features of a computer-generated representation of a subject based on the performance. A method may include obtaining input data corresponding to a pose performed by the subject, the input data including position information defining positions on a face of the subject. The method may further include obtaining an animation model for the subject that includes adjustable controls that control the animation model to define facial features of the computer-generated representation of the face, and matching one or more of the positions on the face with one or more corresponding positions on the animation model. The matching includes using an objective function to project an error onto a control space of the animation model. The method may further include determining, using the projected error and one or more constraints on the adjustable controls, one or more values for one or more of the adjustable controls. The values are configured to control the animation model to cause the computer-generated representation to perform a representation of the pose using the one or more adjustable controls.

Determining control values of an animation model using performance capture

In general, one or more aspects of the subject matter described in this specification can include associating with each clip in a sequence of one or more clips a copy of a three dimensional (3D) scene that was used to create the clip, where the clip is a sequence of one or more images that depict the clip's respective 3D scene from the perspective of one or more virtual cameras. Input identifying a clip in the sequence is received. In response to the receiving, a copy of the identified clip's associated copy of the 3D scene is presented in an editor.

Reconstituting 3D scenes for retakes

Growing community of inventors

Mill Valley, CA, United States of America

Jeffery Bruce Yost