Facebook, Inc.

Facebook Technologies, LLC

Evgeny Zatepyakin

Yauheni Neuhen

Tong Xiao

Peihong Guo

Shu Liang

Sidi Fu

Mengqian Liu

Disclosed herein includes a system, a method, and a non-transitory computer readable medium for rendering a three-dimensional (3D) model of an avatar according to an audio stream including a vocal output of a person and image data capturing a face of the person. In one aspect, phonemes of the vocal output are predicted according to the audio stream, and the predicted phonemes of the vocal output are translated into visemes. In one aspect, a plurality of blendshapes and corresponding weights are determined, according to the corresponding image data of the face, to form the 3D model of the avatar of the person. The visemes may be combined with the 3D model of the avatar to form a 3D representation of the avatar, by synchronizing the visemes with the 3D model of the avatar in time.

System and method for rendering three dimensional face model based on audio stream and image data

Exemplary embodiments relate to the application of media effects such as facial mask overlays, to visual data (such as a video or photo). Publicly-available images may be found and mapped to a mask. In the mapping process, a user may type in the name of a celebrity or public figure, and a system may perform a public image search. In some embodiments, candidate images may be filtered in order to remove images unsuitable for use in masks. Typically, only a single forward-facing image is required for mapping. However, multiple images may be used to provide different angles and allow the user to turn their head while the mask is applied. Mask generation may involve: extracting facial features from the image; mapping the facial features to the user's video; blending/recoloring of either or both of the image or the person's face; and applying the mask in real-time/on the fly.

Methods and systems for identifying target images for a media effect

Exemplary embodiments relate to applications for facial recognition technology and facial overlays to provide gesture-based music track generation. Facial detection technology may be used to analyze a video, to detect a face, and to track the face as a whole (and/or individual features of the face). The features may include, e.g., the locations of the mouth, direction of the eyes, whether the user is blinking, the location of the head in three dimensional space, the movement of the head, etc. Expressions and emotions may also be tracked. Features/expressions/emotions meeting certain conditions may trigger an event, where events may cause a predetermined musical element to play (e.g., drum beat, piano note, guitar chord, etc.). The sum total of the musical elements played may result in the creation of a musical track. The application of events may be balanced based on musical metrics in order to provide a fluent sound.

Methods and systems for playing musical elements based on a tracked face or facial feature

A face tracking system generates a model for extracting a set of facial anchor points on a face within a portion of a face image based a multiple-level cascade of decision trees. The face tracking system identifies a mesh shape adjusted to an image of a face. For each decision tree, the face tracking system identifies an adjustment vector for the mesh shape relative to the image of the face. For each cascade level, the face tracking system combines the identified adjustment for each decision tree to determine a combined adjustment vector for the cascade level. The face tracking system modifies adjustment of the mesh shape to the face in the image based on the combined adjustment vector. The face tracking system reduces the model to a dictionary and atom weights using a learned dictionary. The model may be more easily transmitted to devices and stored on devices.

Growing community of inventors

London, United Kingdom

Evgeny Zatepyakin