Facebook Technologies, LLC

Disney Enterprises, Inc.

Eth Zurich (eidgenoessische Technische Hochschule Zurich)

Eth Zurich (eidgenossische Technische Hochschule Zurich)

Chenglei Wu

Derek Edward Bradley

Thabo D Beeler

Jason Saragih

Takaaki Shiratori

Tomas Simon Kreuz

Shoou-I Yu

Yaser Sheikh

Stephen Anthony Lombardi

Markus H Gross

In one embodiment, a system may access a codec that encodes an appearance associated with a subject and comprise codec portions that respectively correspond to body parts of the subject. The system may generate a training codec that comprises a first subset of the codec portions (a first set of body parts) and a modified second subset of the codec portions (muted body parts). The system may decode the training codec using a machine-learning model to generate a mesh of the subject. The system may transform the mesh of the subject based on a predetermined pose. The system may update the machine-learning model based on a comparison between the transformed mesh and a target mesh of the subject having the predetermined pose. The system in the present application can train a machine-learning model to render an avatar with a pose using uncorrelated codec portions corresponding to different body parts.

Systems and methods for rendering avatar using uncorrelated codec portions in codec

The disclosed computer system may include an input module, an autoencoder, and a rendering module. The input module may receive geometry information and images of a subject. The geometry information may be indicative of variation in geometry of the subject over time. Each image may be associated with a respective viewpoint and may include a view-dependent texture map of the subject. The autoencoder may jointly encode texture information and the geometry information to provide a latent vector. The autoencoder may infer, using the latent vector, an inferred geometry and an inferred view-dependent texture of the subject for a predicted viewpoint. The rendering module may be configured to render a reconstructed image of the subject for the predicted viewpoint using the inferred geometry and the inferred view-dependent texture. Various other systems and methods are also disclosed.

Systems and methods for rendering avatars with deep appearance models

Multiple cameras with different orientations capture images of an object positioned at a target position relative to the cameras. Images from each camera are processed in parallel to determine depth information from correspondences between different regions within an image captured by each image capture device in parallel. Depth information for images from each camera is modified in parallel based on shading information for the images and stereoscopic information from the images. In various embodiments, the depth information is refined by minimizing a total energy from intensities of portions of the images having a common depth and intensities of portions of the image from shading information from images captured by multiple cameras. The modified depth information from multiple images is combined to generate a reconstruction of the object positioned at the target position.

Generating a representation of an object from depth information determined in parallel from images captured by multiple cameras

A system and method for non-invasive reconstruction of an entire object-specific or person-specific teeth row from just a set of photographs of the mouth region of an object (e.g., an animal) or a person (e.g., an actor or a patient) are provided. A teeth statistic model defining individual teeth in a teeth row can be developed. The teeth statistical model can jointly describe shape and pose variations per tooth, and as well as placement of the individual teeth in the teeth row. In some embodiments, the teeth statistic model can be trained using teeth information from 3D scan data of different sample subjects. The 3D scan data can be used to establish a database of teeth of various shapes and poses. Geometry information regarding the individual teeth can be extracted from the 3D scan data. The teeth statistic model can be trained using the geometry information regarding the individual teeth.

Model-based teeth reconstruction

Techniques and systems are described for generating an anatomically-constrained local model and for performing performance capture using the model. The local model includes a local shape subspace and an anatomical subspace. In one example, the local shape subspace constrains local deformation of various patches that represent the geometry of a subject's face. In the same example, the anatomical subspace includes an anatomical bone structure, and can be used to constrain movement and deformation of the patches globally on the subject's face. The anatomically-constrained local face model and performance capture technique can be used to track three-dimensional faces or other parts of a subject from motion data in a high-quality manner. Local model parameters that best describe the observed motion of the subject's physical deformations (e.g., facial expressions) under the given constraints are estimated through optimization. The optimization can solve for rigid local patch motion, local patch deformation, and the rigid motion of the anatomical bones. The solution can be formulated as an energy minimization problem for each frame that is obtained for performance capture.

Methods and systems of generating an anatomically-constrained local model for performance capture

Methods and systems of performing performance capture using an anatomically-constrained local model

Growing community of inventors

Pittsburgh, PA, United States of America

Chenglei Wu