Apple Inc.

Stefan Brugger

Brian Amberg

Sofien Bouaziz

Thibaut Weise

Reinhard Knothe

Thomas Kiser

Patrick Snape

Alex Mansfield

Sebastian Martin

Ali Benlalah

Michael Tschannen

Anna Volokitin

Advances in deep generative models (DGM) have led to the development of neural face video compression codecs that are capable of using an order of magnitude less data than 'traditional' engineered codecs. These “neural” codecs can reconstruct a target image by warping a source image to approximate the content of the target image and using a DGM to compensate for imperfections in the warped source image. The determined warping operation may be encoded and transmitted using less data (e.g., transmitting a small number of keypoints, rather than a dense flow field), leading to the bandwidth savings compared to traditional codecs. However, by relying on a single source image only, these methods can lead to inaccurate reconstructions. The techniques presented herein improve image reconstruction quality while maintaining bandwidth savings, via a combination of using multiple source images (i.e., containing multiple views of the first human subject) and novel feature aggregation techniques.

Neural face video compression using multiple views

A three-dimensional model (e.g., motion capture model) of a user is generated from captured images or captured video of the user. A machine learning network may track poses and expressions of the user to generate and refine the three-dimensional model. Refinement of the three-dimensional model may provide more accurate tracking of the user's face. Refining of the three-dimensional model may include refining the determinations of poses and expressions at defined locations (e.g., eye corners and/or nose) in the three-dimensional model. The refining may occur in an iterative process. Tracking of the three-dimensional model over time (e.g., during video capture) may be used to generate an animated three-dimensional model (e.g., an animated puppet) of the user that simulates the user's poses and expressions.

Generating animated three-dimensional models from captured images

Growing community of inventors

Basel Land, Switzerland

Stefan Brugger