Meta Platforms Technologies, LLC

Meta Platforms, Inc.

Yaniv Nechemia Taigman

Oran Gafni

Lior Wolf

Adam Polyak

In one embodiment, a method includes accessing a text input and a scene input corresponding to the text input, wherein the scene input comprises semantic segmentations, generating text tokens for the text input and scene tokens for the scene input by machine-learning models, generating predicted image tokens based on the text tokens and the scene tokens by the machine-learning models, and generating an image corresponding to the text input and the scene input based on the predicted image tokens by the machine-learning models.

Scene-based text-to-image generation with human priors

A video generation system is described that extracts one or more characters or other objects from a video, re-animates the character, and generates a new video in which the extracted characters. The system enables the extracted character(s) to be positioned and controlled within a new background scene different from the original background scene of the source video. In one example, the video generation system comprises a pose prediction neural network having a pose model trained with (i) a set of character pose training images extracted from an input video of the character and (ii) a simulated motion control signal generated from the input video. In operation, the pose prediction neural network generates, in response to a motion control input from a user, a sequence of images representing poses of a character. A frame generation neural network generates output video frames that render the character within a scene.

Growing community of inventors

Raanana, Israel

Yaniv Nechemia Taigman