Scene, pose, and environment replacement in videos and images using generative ai

Abstract

Systems and methods for replacement of scene, pose, and environment in videos and images using generative AI are provided. An example method includes receiving a first image of a face of a person, a second image of a body adopting a pose, and a text including a description of an environment of a scene; encoding the first image into an image embedding; extracting, from the second image, information concerning the pose of the body; encoding the text into a text embedding; randomly generating a noise for the image embedding; combining the noise and the image embedding to obtain a noisy image embedding; providing the noisy image embedding, the text embedding, and the information concerning the pose to a neural network to obtain a second noise; removing the second noise from the noisy image embedding to obtain a denoised image embedding; and decoding the denoised image embedding into an output image.