Adobe, Inc.

Jun Saito

Jimei Yang

Ruben Eduardo Villegas

Duygu Ceylan Aksit

Yang Zhou

Vladimir Kim

Dingzeyu Li

James Acquavella

Deepali Aneja

David Werner

Elya Shechtman

Baldo Antonio Faieta

Aaron Phillip Hertzmann

Shabnam Ghadar

Noam Aigerman

Alex Charles Filipkowski

Siddhartha Chaudhuri

Omid Poursaeed

Jinrong Xie

Thibault Groueix

Elnaz Morad

Kunal Gupta

Nitin Saini

Wolfram Sebastian Starke

Embodiments are disclosed for generating a gesture reenactment video sequence corresponding to a target audio sequence using a trained network based on a video motion graph generated from a reference speech video. In particular, in one or more embodiments, the disclosed systems and methods comprise receiving a first input including a reference speech video and generating a video motion graph representing the reference speech video, where each node is associated with a frame of the reference video sequence and reference audio features of the reference audio sequence. The disclosed systems and methods further comprise receiving a second input including a target audio sequence, generating target audio features, identifying a node path through the video motion graph based on the target audio features and the reference audio features, and generating an output media sequence based on the identified node path through the video motion graph paired with the target audio sequence.

Generating gesture reenactment video from video motion graphs using machine learning

Methods, systems, and non-transitory computer readable storage media are disclosed for generating digital chain pull paintings in digital images. The disclosed system generate, utilizing a neural network, a plurality of matrices over an ambient space for a plurality of polygons of a three-dimensional mesh based on a plurality of features of the plurality of polygons associated with the three-dimensional mesh. The disclosed system determines a gradient field based on the plurality of matrices of the plurality of polygons. The disclosed system generates a mapping for the three-dimensional mesh based on the gradient field and a differential operator corresponding to the three-dimensional mesh.

Generating mappings of three-dimensional meshes utilizing neural network gradient fields

Methods, systems, and non-transitory computer readable storage media are disclosed for utilizing unsupervised learning of discrete human motions to generate digital human motion sequences. The disclosed system utilizes an encoder of a discretized motion model to extract a sequence of latent feature representations from a human motion sequence in an unlabeled digital scene. The disclosed system also determines sampling probabilities from the sequence of latent feature representations in connection with a codebook of discretized feature representations associated with human motions. The disclosed system converts the sequence of latent feature representations into a sequence of discretized feature representations by sampling from the codebook based on the sampling probabilities. Additionally, the disclosed system utilizes a decoder to reconstruct a human motion sequence from the sequence of discretized feature representations. The disclosed system also utilizes a reconstruction loss and a distribution loss to learn parameters of the discretized motion model.

Generating human motion sequences utilizing unsupervised learning of discretized features via a neural network encoder-decoder

One example method involves a processing device that performs operations that include receiving a request to retarget a source motion into a target object. Operations further include providing the target object to a contact-aware motion retargeting neural network trained to retarget the source motion into the target object. The contact-aware motion retargeting neural network is trained by accessing training data that includes a source object performing the source motion. The contact-aware motion retargeting neural network generates retargeted motion for the target object, based on a self-contact having a pair of input vertices. The retargeted motion is subject to motion constraints that: (i) preserve a relative location of the self-contact and (ii) prevent self-penetration of the target object.

Contact-aware retargeting of motion

Digital object animation techniques are described. In a first example, translation-based animation of the digital object operates using control points of the digital object. In another example, the animation system is configured to minimize an amount of feature positions that are used to generate the animation. In a further example, an input pose is normalized through use of a global scale factor to address changes in a z-position of a subject in different digital images. Yet further, a body tracking module is used to computing initial feature positions. The initial feature positions are then used to initialize a face tracker module to generate feature positions of the face. The animation system also supports a plurality of modes used to generate the digital object, techniques to define a base of the digital object, and a friction term limiting movement of features positions based on contact with a ground plane.

Digital object animation using control points

Embodiments are disclosed for re-timing a video sequence to an audio sequence based on the detection of motion beats in the video sequence and audio beats in the audio sequence. In particular, in one or more embodiments, the disclosed systems and methods comprise receiving a first input, the first input including a video sequence, detecting motion beats in the video sequence, receiving a second input, the second input including an audio sequence, detecting audio beats in the audio sequence, modifying the video sequence by matching the detected motions beats in the video sequence to the detected audio beats in the audio sequence, and outputting the modified video sequence.

Re-timing a video sequence to an audio sequence based on motion and audio beat detection

Motion retargeting with kinematic constraints is implemented in a digital medium environment. Generally, the described techniques provide for retargeting motion data from a source motion sequence to a target visual object. Accordingly, the described techniques position a target visual object in a defined visual environment to identify kinematic constraints of the target object relative to the visual environment. Further, the described techniques utilize an iterative optimization process that fine tunes the conformance of retargeted motion of a target object to the identified kinematic constraints.

Motion retargeting with kinematic constraints

The present disclosure relates to systems, methods, and non-transitory computer readable media for accurately, efficiently, and flexibly identifying and providing digital images of human figures in poses corresponding to a query pose. In particular, the disclosed systems can provide multiple approaches to searching for and providing pose images, including identifying a digital image depicting a human figure in a particular pose based on a query digital image that depicts the pose or identifying a digital image depicting a human figure in a particular pose based on a virtual mannequin. Indeed, the disclosed systems can provide a manipulable virtual mannequin that defines a query pose for searching a repository of digital images. Additionally, the disclosed systems can generate and provide digital pose image groups by clustering digital images together according to poses of human figures within a pose feature space.

Identifying and providing digital images depicting human poses utilizing visual interactive content search and virtual mannequins

Digital character animation automated generation techniques are described that are implemented by an animation generation system via a computing device. These techniques enable the animation generation system to generate an animation of a digital character automatically and without user intervention responsive to a user input of a target action such that the digital character is capable of performing a complex set of actions in a precise and realistic manner within an environment contained within digital content, e.g., an animation as part of a digital video.

Neural state machine digital character animation

The present disclosure relates to systems, non-transitory computer-readable media, and methods for intelligently blending inverse kinematic (IK) solutions to more naturally depict joint positioning and/or movement of digital animated characters. In particular, in one or more embodiments, the character animation system can blend two IK solutions for an elbow joint based on a shoulder angle. For example, the character animation system can utilize a blending region to dynamically blend IK solutions as the shoulder angle moves through the blending region, thereby smoothly modifying bend direction and elbow position of the animated character arm. Based on the modified elbow position relative to a wrist position and a shoulder position, the animated character system can simulate more accurate, natural arm movements while reducing time and interactions needed to generate realistic animation sequences.

Inverse kinematic solution blending in digital character animation

In implementations of object animation using generative neural networks, one or more computing devices of a system implement an animation system for reproducing animation of an object in a digital video. A mesh of the object is obtained from a first frame of the digital video and a second frame of the digital video having the object is selected. Features of the object from the second frame are mapped to vertices of the mesh, and the mesh is warped based on the mapping. The warped mesh is rendered as an image by a neural renderer and compared to the object from the second frame to train a neural network. The rendered image is then refined by a generator of a generative adversarial network which includes a discriminator. The discriminator trains the generator to reproduce the object from the second frame as the refined image.

Growing community of inventors

Seattle, WA, United States of America

Jun Saito