Meta Platforms, Inc.

Facebook Technologies, LLC

Meta Platforms Technologies, LLC

Oran Gafni

Lior Wolf

Yaniv Nechemia Taigman

Yaniv Taigman

Peizhao Zhang

Thibault Michel Max Peyronel

Camille Couprie

Maxime Mohamad Oquab

Pierre Stock

Daniel Raynald David Haziza

Tao Xu

Oron Ashual

Onur Çelebi

Patrick Labatut

Adam Polyak

A computing device performs generating a first identity encoding representing a first facial identity of the person based on an image of a person, generating a second identity encoding representing a second facial identity different from the first facial identity of the person based on the first identity encoding, generating a source encoding by using an encoder to process a source image of the person having an expression, generating an intermediate image by using a decoder to process the source encoding and the second identity encoding, the intermediate image including a face having the second facial identity and the expression of the person in the source image, and generating an output image by blending the source image with facial features of the face in the intermediate image.

Altering a facial identity in a video stream

In one embodiment, a method includes accessing a text input and a scene input corresponding to the text input, wherein the scene input comprises semantic segmentations, generating text tokens for the text input and scene tokens for the scene input by machine-learning models, generating predicted image tokens based on the text tokens and the scene tokens by the machine-learning models, and generating an image corresponding to the text input and the scene input based on the predicted image tokens by the machine-learning models.

Scene-based text-to-image generation with human priors

In one embodiment, a first device may receive, from a second device, a reference landmark map identifying locations of facial features of a user of the second device depicted in a reference image and a feature map, generated based on the reference image, representing an identity of the user. The first device may receive, from the second device, a current compressed landmark map based on a current image of the user and decompress the current compressed landmark map to generate a current landmark map. The first device may update the feature map based on a motion field generated using the reference landmark map and the current landmark map. The first device may generate scaling factors based on a normalization facial mask of pre-determined facial features of the user. The first device may generate an output image of the user by decoding the updated feature map using the scaling factors.

Systems and method for low bandwidth video-chat compression

In one embodiment, a method includes receiving a first image depicting a context including one or more persons having one or more respective poses, receiving a second image depicting a target person having an original pose, where the target person is to be inserted into the context depicted in the first image, generating a target segmentation mask specifying a new pose for the target person in the context of the first image based on the first image, generating a third image depicting the target person having the new pose based on the second image and the target segmentation mask, and generating an output image based on the first image and the third image, the output image depicting the one or more persons having the one or more respective poses and the target person having the new pose.

Context-aware human generation in an image

A video generation system is described that extracts one or more characters or other objects from a video, re-animates the character, and generates a new video in which the extracted characters. The system enables the extracted character(s) to be positioned and controlled within a new background scene different from the original background scene of the source video. In one example, the video generation system comprises a pose prediction neural network having a pose model trained with (i) a set of character pose training images extracted from an input video of the character and (ii) a simulated motion control signal generated from the input video. In operation, the pose prediction neural network generates, in response to a motion control input from a user, a sequence of images representing poses of a character. A frame generation neural network generates output video frames that render the character within a scene.

Controllable video characters with natural motions extracted from real-world videos

In one embodiment, a method includes generating a keypoint pose and a dense pose for a first person in a first pose based on a first image comprising the first person in the first pose, generating an input semantic segmentation map corresponding to a second person in a second pose based on a second image comprising the second person in the second pose, generating a target semantic segmentation map corresponding to the second person in the first pose by processing the keypoint pose, the dense pose, and the input segmentation map using a first machine-learning model, generating an encoding vector representing the second person based on the second image, and generating a target image of the second person in the first pose by processing the encoding vector and the target segmentation map using a second machine-learning model.

Motion transfer using machine-learning models

Growing community of inventors

Ramat Gan, Israel

Oran Gafni