Pinscreen, Inc.

Adobe, Inc.

University of Southern California

Liwen Hu

Hao Li

Lingyu Wei

Shunsuke Saito

Koki Nagano

Jaewoo Seo

Huiwen Luo

Zejian Wang

Jimei Yang

Byungmoon Kim

Zhili Chen

Jens Fursund

Duygu Ceylan

Cosimo Wei

A system, method, and apparatus for generating a normalized three-dimensional model of a human face from a single unconstrained two-dimensional image of the human face. The system includes a processor that executes instructions including receiving the single unconstrained two-dimensional image of the human face, using an inference network to determine an inferred normalized three-dimensional model of the human face based on the single unconstrained two-dimensional image of the human face, and using a refinement network to iteratively determine the normalized three-dimensional model of the human face with a neutral expression and unshaded albedo textures under diffuse lighting conditions based on the inferred normalized three-dimensional model of the human face.

Normalized three-dimensional avatar synthesis and perceptual refinement

A system, method, and apparatus for generating a normalization of a single two-dimensional image of an unconstrained human face. The system receives the single two-dimensional image of the unconstrained human face, generates an undistorted face based on the unconstrained human face by removing perspective distortion from the unconstrained human face via a perspective undistortion network, generates an evenly lit face based on the undistorted face by normalizing lighting of the undistorted face via a lighting translation network, and generates a frontalized and neutralized expression face based on the evenly lit face via an expression neutralization network.

Normalization of facial images using deep neural networks

Devices and methods for single-view 3D hair modeling are disclosed. The method for single-view 3D hair modeling includes training, by a neural network processor, a volumetric autoencoder to encode a plurality of 3D hairstyles into latent features, and to generate an output based on the latent features. The method for single-view 3D hair modeling includes training, by the neural network processor, an embedding network to determine hair coefficients of a single hairstyle from an input image. The method for single-view 3D hair modeling includes receiving, by the neural network processor, the input image. The method for single-view 3D hair modeling includes synthesizing, by the neural network processor, hair strands to generate a single-view 3D model of the single hairstyle based on the volumetric autoencoder, the embedding network, and the input image.

3D hair synthesis using volumetric variational autoencoders

A system for generating three-dimensional facial models including photorealistic hair and facial textures includes creating a facial model with reliance upon neural networks based upon a single two-dimensional input image. The photorealistic hair is created by finding a subset of similar three-dimensional polystrip hairstyles from a large database of polystrip hairstyles, selecting the most-alike polystrip hairstyle, deforming that polystrip hairstyle to better fit the hair of the two-dimensional image. Then, collisions and bald spots are corrected, and suitable textures are applied. Finally, the facial model and polystrip hairstyle are combined into a final three-dimensional avatar.

Avatar digitization from a single image for real-time rendering

A method for generating three-dimensional facial models and photorealistic textures from inferences using deep neural networks relies upon generating a low frequency and a high frequency albedo map of the full and partial face, respectively. Then, the high frequency albedo map may be used for comparison with correlation matrices generated by a neural network trained by a large scale, high-resolution facial dataset with simulated partial visibility. The corresponding correlation matrices of the complete facial textures can then be retrieved. Finally, a full facial texture map may be synthesized, using convex combinations of the correlation matrices. A photorealistic facial texture for the three-dimensional face rendering can be obtained through optimization using the deep neural network and a loss function that incorporates the blended target correlation matrices.

Photorealistic facial texture inference using deep neural networks

Certain embodiments involve recognizing combinations of body shape, pose, and clothing in three-dimensional input images. For example, synthetic training images are generated based on user inputs. These synthetic training images depict different training figures with respective combinations of a body pose, a body shape, and a clothing item. A machine learning algorithm is trained to recognize the pose-shape-clothing combinations in the synthetic training images and to generate feature descriptors describing the pose-shape-clothing combinations. The trained machine learning algorithm is outputted for use by an image manipulation application. In one example, an image manipulation application uses a feature descriptor, which is generated by the machine learning algorithm, to match an input figure in an input image to an example image based on a correspondence between a pose-shape-clothing combination of the input figure and a pose-shape-clothing combination of an example figure in the example image.

Growing community of inventors

Los Angeles, CA, United States of America

Liwen Hu