Apple Inc.

Brian Amberg

Thibaut Weise

Stefan Brugger

Aurelio Guzmán

Nicolas V Scapel

Hoan K Pham

Guillaume P Barlier

Thomas Goossens

Bruno M Sommer

Sofien Bouaziz

Alexandre R Moha

Justin D Stoyles

Reinhard Knothe

Thomas Kiser

Nina Damasky

Patrick Snape

Alex Mansfield

Daniel R Kurz

Yanghai Tsin

Peter Kaufmann

Sebastian Martin

Ali Benlalah

Michael Tschannen

Anna Volokitin

Advances in deep generative models (DGM) have led to the development of neural face video compression codecs that are capable of using an order of magnitude less data than 'traditional' engineered codecs. These “neural” codecs can reconstruct a target image by warping a source image to approximate the content of the target image and using a DGM to compensate for imperfections in the warped source image. The determined warping operation may be encoded and transmitted using less data (e.g., transmitting a small number of keypoints, rather than a dense flow field), leading to the bandwidth savings compared to traditional codecs. However, by relying on a single source image only, these methods can lead to inaccurate reconstructions. The techniques presented herein improve image reconstruction quality while maintaining bandwidth savings, via a combination of using multiple source images (i.e., containing multiple views of the first human subject) and novel feature aggregation techniques.

Neural face video compression using multiple views

Various implementations disclosed herein include devices, systems, and methods that use event camera data to track deformable objects such as faces, hands, and other body parts. One exemplary implementation involves receiving a stream of pixel events output by an event camera. The device tracks the deformable object using this data. Various implementations do so by generating a dynamic representation of the object and modifying the dynamic representation of the object in response to obtaining additional pixel events output by the event camera. In some implementations, generating the dynamic representation of the object involves identifying features disposed on the deformable surface of the object using the stream of pixel events. The features are determined by identifying patterns of pixel events. As new event stream data is received, the patterns of pixel events are recognized in the new data and used to modify the dynamic representation of the object.

Deformable object tracking

Systems and methods for generating a video of an emoji that has been puppeted using inputs from image, depth, and audio. The inputs can capture facial expressions of a user, eye, eyebrow, mouth, and head movements. A pose, held by the user, can be detected that can be used to generate supplemental animation. The emoji can further be animated using physical properties associated with the emoji and captured movements. An emoji of a dog can have its ears move in response to an up-and-down movement, or a shaking of the head. The video can be sent in a message to one or more recipients. A sending device can render the puppeted video in accordance with hardware and software capabilities of a recipient's computer device.

Animated representation of facial expression

A three-dimensional model (e.g., motion capture model) of a user is generated from captured images or captured video of the user. A machine learning network may track poses and expressions of the user to generate and refine the three-dimensional model. Refinement of the three-dimensional model may provide more accurate tracking of the user's face. Refining of the three-dimensional model may include refining the determinations of poses and expressions at defined locations (e.g., eye corners and/or nose) in the three-dimensional model. The refining may occur in an iterative process. Tracking of the three-dimensional model over time (e.g., during video capture) may be used to generate an animated three-dimensional model (e.g., an animated puppet) of the user that simulates the user's poses and expressions.

Generating animated three-dimensional models from captured images

Emojicon puppeting

Growing community of inventors

Zurich, Switzerland

Brian Amberg