Snap Inc.

Maxim Lukin

Aleksandr Mashrabov

Pavel Savchenkov

Nikolai Smirnov

Yurii Volkov

Ivan Belonogov

Systems and methods for text and audio-based real-time face reenactment are provided. An example method includes receiving an input text and a target image, where the target image including a target face, generating, based on the input text, a sequence of sets of acoustic features corresponding to the input text, generating, based on the sequence of sets of acoustic features, a sequence of sets of mouth key points, generating, based on the sequence of sets of mouth key points, a sequence of sets of facial key points, determining, based on the set sequence of sets of facial key points, a sequence of deformations of the target face, and applying the sequence of deformations to the target image, thereby generating a sequence of frames of an output video.

Text and audio-based real-time face reenactment

Provided are systems and methods for text and audio-based real-time face reenactment. An example method includes receiving an input text and a target image, the target image including a target face; generating, based on the input text, a sequence of sets of acoustic features representing the input text; generating, based on the sequence of sets of acoustic features, a sequence of sets of mouth key points; generating, based on the sequence of sets of mouth key points, a sequence of sets of facial key points; generating, by the computing device and based on the sequence of sets of the facial key points and the target image, a sequence of frames; and generating, based on the sequence of frames, an output video. Each of the frames includes the target face modified based on at least one set of mouth key points of the sequence of sets of mouth key points.

Provided are systems and methods for realistic head turns and face animation synthesis. An example method may include receiving frames of a source video with the head and the face of a source actor. The method may then proceed with generating sets of source pose parameters that represent positions of the head and facial expressions of the source actor. The method may further include receiving at least one target image including the target head and the target face of a target person, determining target identity information associated with the target face, and generating an output video based on the target identity information and the sets of source pose parameters. Each frame of the output video can include an image of the target face modified to mimic at least one of the positions of the head of the source actor and at least one of facial expressions of the source actor.

Systems and methods for realistic head turns and face animation synthesis on mobile device

Provided are systems and methods for text and audio-based real-time face reenactment. An example method includes receiving an input text and a target image, the target image including a target face; generating, based on the input text, a sequence of sets of acoustic features representing the input text; determining, based on the sequence of sets of acoustic features, a sequence of sets of scenario data indicating modifications of the target face for pronouncing the input text; generating, based on the sequence of sets of scenario data, a sequence of frames, wherein each of the frames includes the target face modified based on at least one of the sets of scenario data; generating, based on the sequence of frames, an output video; and synthesizing, based on the sequence of sets of acoustic features, an audio data and adding the audio data to the output video.

Growing community of inventors

Sochi, Russia

Maxim Lukin