Apple Inc.

Andrew P Mason

Olivier Soares

Fletcher R Rothkopf

Grant H Mulliken

Izzet B Yildiz

Lilli I Jonsson

Graham L Fyffe

Akiko Ikkai

Haarm-Pieter Duiker

John S McCarten

Thomas Gebauer

Rendering an avatar for a user in a communication session includes obtaining one or more identity textures for the user for the session and obtaining, throughout the session, shading latents for the user. The shading latents are derived from lighting and other information and are utilized as input into a decoder to obtain one or more neural maps. A target texture is generated by warping the identity textures based on the neural maps. The target texture is used for rendering the avatar.

Neural shading

Rendering an avatar may include determining an expression to be represented by an avatar, obtaining a blood texture map associated with the expression, wherein the blood texture map represents an offset of coloration from an albedo map for the expression, and rendering the avatar utilizing the blood texture map.

Machine learning-based blood flow tracking

Sensor data indicating a user's response to an avatar experience in which the user experiences a rendered avatar model is obtained. A perceptual quality metric value corresponding to the rendered avatar model is determined based on the sensor data and a determined relationship between the sensor data and the perceptual quality metric value. The avatar model is re-rendered for display based on the perceptual quality metric value.

Perceptually and physiologically constrained optimization of avatar models

Rendering an avatar for a user in a communication session includes obtaining latent variables for expression and identity of a face, applying a concatenation of the latent variables to an expression decoder of a trained asymmetric joint autoencoder for expression and identification to obtain an expression mesh of a face, and rendering an avatar using the expression mesh of the face. The asymmetric joint autoencoder includes two encoders and two decoders, where an expression decoder utilizes the concatenated latents and the identity decoder utilizes the identity latents and not the expression latents.

Joint autoencoder for identity and expression

Rendering an avatar in a selected environment may include determining as inputs into an inferred shading network, an expression geometry to be represented by an avatar, head pose, and camera angle, along with a lighting representation for the selected environment. The inferred shading network may then generate a texture of a face to be utilized in rendering the avatar. The lighting representation may be obtained as lighting latent variables which are obtained from an environment autoencoder trained on environment images with various lighting conditions.

Inferred shading

Techniques are disclosed for providing an avatar personalized for a specific person based on known data from a relatively large population of individuals and a relatively small data sample of the specific person. Auto-encoder neural networks are used in a novel manner to capture latent-variable representations of facial models. Once such models are developed, a very limited data sample of a specific person may be used in combination with convolutional-neural-networks or statistical filters, and driven by audio/visual input during real-time operations, to generate a realistic avatar of the specific individual's face. In some embodiments, conditional variables may be encoded (e.g. gender, age, body-mass-index, ethnicity, emotional state). In other embodiments, different portions of a face may be modeled separately and combined at run-time (e.g., face, tongue and lips). Models in accordance with this disclosure may be used to generate resolution independent output.

Growing community of inventors

Cottesloe, Australia

Andrew P Mason