Video reenactment taking into account temporal information

Abstract

Apparati, methods, and computer readable media for inserting identity information from a source image (static image or video) () into a destination video (), while mimicking motion of the destination video (). In an apparatus embodiment, an identity encoder () is configured to encode identity information of the source image (). When source image () is a multi-frame static image or a video, an identity code aggregator () is positioned at an output of the identity encoder (), and produces an identity vector (). A driver encoder () is coupled to the destination (driver) video (), and has two components: a pose encoder () configured to encode pose information of the destination video (), and a motion encoder () configured to separately encode motion information of the destination video (). The driver encoder () produces two vectors: a pose vector () and a motion vector (). A neural network generator () has three inputs: the identity vector (), the pose vector (), and the motion vector (). The neural network generator () is configured to generate, in response to these three inputs, a composite video () comprising identity information of the source image () inserted into the destination video (), where the composite video () has substantially the same temporal information as the destination video ().