Adobe, Inc.

University of Massachusetts

Aaron Phillip Hertzmann

Stephen J DiVerdi

Sylvain Philippe Paris

Elya Shechtman

Jimei Yang

Vladimir Kim

Mehmet Yumer

Bryan Russell

Cuong D Nguyen

Holger Winnemoeller

Wilmot Wei-Mau Li

Jun Xie

Brian Lynn Price

Matthew David Fisher

Lubomira A Dontcheva

Richard Zhang

Alan Wilson

Junyan Zhu

Zhicheng Liu

Ian D Sachs

Jian Zhao

Jacob Minyoung Huh

Xiaoyong Shen

Ana Belén Serrano Pacheu

Erik Andreas Härkönen

Davis Rempe

Hailin Jin

Jianming Zhang

Zhaowen Wang

Aseem Agarwala

Chen Fang

Yijun Li

Feng Liu

Yuyan Song

Jun Saito

Ruben Eduardo Villegas

Brian David Williams

Zoya Bylinskii

Duygu Ceylan Aksit

Evangelos Kalogerakis

Seth Shaw

Shayan Hoshyari

Difan Liu

Saining Xie

Mohammad Amin Sadeghi

Shuhui Jiang

Mohammad Rastegari

Jianan Li

Manuel Rodriguez Ladron De Guevara

Chenxi Liu

Li Yi

Methods, systems, and non-transitory computer readable storage media are disclosed for utilizing a multi-stroke neural network for modifying a digital image via a plurality of generated stroke parameters in a single pass of the neural network. Specifically, the disclosed system utilizes an encoder neural network to generate an encoding of a digital image. The disclosed system then utilizes a decoder neural network that generates a sequence of stroke parameters for digital drawing strokes from the encoding in a single pass of the encoder neural network and decoder neural network. Additionally, the disclosed system utilizes a renderer neural network to render the digital drawing strokes on a digital canvas according to the sequence of stroke parameters. In additional embodiments, the disclosed system utilizes a balance of loss functions to learn parameters of the multi-stroke neural network to generate stroke parameters according to various rendering styles.

Generating stylized digital images via drawing stroke optimization utilizing a multi-stroke neural network

A generative neural network control system controls a generative neural network by modifying the intermediate latent space in the generative neural network. The generative neural network includes multiple layers each generating a set of activation values. An initial layer (and optionally additional layers) receives an input latent vector, and a final layer outputs an image generated based on the input latent vector. The data that is input to each layer (other than the initial layer) is referred to as data in an intermediate latent space. The data in the intermediate latent space includes activation values (e.g., generated by the previous layer or modified using various techniques) and optionally a latent vector. The generative neural network control system modifies the intermediate latent space to achieve various different effects when generating a new image.

Controlling a neural network through intermediate latent spaces

One example method involves a processing device that performs operations that include receiving a request to retarget a source motion into a target object. Operations further include providing the target object to a contact-aware motion retargeting neural network trained to retarget the source motion into the target object. The contact-aware motion retargeting neural network is trained by accessing training data that includes a source object performing the source motion. The contact-aware motion retargeting neural network generates retargeted motion for the target object, based on a self-contact having a pair of input vertices. The retargeted motion is subject to motion constraints that: (i) preserve a relative location of the self-contact and (ii) prevent self-penetration of the target object.

Contact-aware retargeting of motion

Embodiments are disclosed for computing accurate smooth occluding contours. In one embodiment, a method of computing accurate smooth occluding contours includes projecting a boundary polygon associated with a first region of a three-dimensional (3D) object to a two-dimensional (2D) image plane, the boundary polygon comprising a plurality of contour vertices and edges connecting the plurality of contour vertices, triangulating the first region in the 2D image plane to generate a 2D triangulation, and generating a 3D mesh for the first region by mapping the 2D triangulation to the 3D object.

Accurate smooth occluding contours

Techniques for generating a stylized drawing of three-dimensional (3D) shapes using neural networks are disclosed. A processing device generates a set of vector curve paths from a viewpoint of a 3D shape; extracts, using a first neural network of a plurality of neural networks of a machine learning model, surface geometry features of the 3D shape based on geometric properties of surface points of the 3D shape; determines, using a second neural network of the plurality of neural networks of the machine learning model, a set of at least one predicted stroke attribute based on the surface geometry features and a predetermined drawing style; generates, based on the at least one predicted stroke attribute, a set of vector stroke paths corresponding to the set of vector curve paths; and outputs a two-dimensional (2D) stylized stroke drawing of the 3D shape based at least on the set of vector stroke paths.

Generation of stylized drawing of three-dimensional shapes using neural networks

The present disclosure includes methods and systems for identifying and manipulating a segment of a three-dimensional digital model based on soft classification of the three-dimensional digital model. In particular, one or more embodiments of the disclosed systems and methods identify a soft classification of a digital model and utilize the soft classification to tune segmentation algorithms. For example, the disclosed systems and methods can utilize a soft classification to select a segmentation algorithm from a plurality of segmentation algorithms, to combine segmentation parameters from a plurality of segmentation algorithms, and/or to identify input parameters for a segmentation algorithm. The disclosed systems and methods can utilize the tuned segmentation algorithms to accurately and efficiently identify a segment of a three-dimensional digital model.

Utilizing soft classifications to select input parameters for segmentation algorithms and identify segments of three-dimensional digital models

In some embodiments, a model training system obtains a set of animation models. For each of the animation models, the model training system renders the animation model to generate a sequence of video frames containing a character using a set of rendering parameters and extracts joint points of the character from each frame of the sequence of video frames. The model training system further determines, for each frame of the sequence of video frames, whether a subset of the joint points are in contact with a ground plane in a three-dimensional space and generates contact labels for the subset of the joint points. The model training system trains a contact estimation model using training data containing the joint points extracted from the sequences of video frames and the generated contact labels. The contact estimation model can be used to refine a motion model for a character.

Motion model refinement based on contact analysis and optimization

A target image is projected into a latent space of generative model by determining a latent vector by applying a gradient-free technique and a class vector by applying a gradient-based technique. An image is generated from the latent and class vectors, and a loss function is used to determine a loss between the target image and the generated image. This determining of the latent vector and the class vector, generating an image, and using the loss function is repeated until a loss condition is satisfied. In response to the loss condition being satisfied, the latent and class vectors that resulted in the loss condition being satisfied are identified as the final latent and class vectors, respectively. The final latent and class vectors are provided to the generative model and multiple weights of the generative model are adjusted to fine-tune the generative model.

Projecting images to a generative model based on gradient-free latent vector determination

Mixing segmentation algorithms utilizing soft classifications to identify segments of three-dimensional digital models

In some embodiments, a motion model refinement system receives an input video depicting a human character and an initial motion model describing motions of individual joint points of the human character in a three-dimensional space. The motion model refinement system identifies foot joint points of the human character that are in contact with a ground plane using a trained contact estimation model. The motion model refinement system determines the ground plane based on the foot joint points and the initial motion model and constructs an optimization problem for refining the initial motion model. The optimization problem minimizes the difference between the refined motion model and the initial motion model under a set of plausibility constraints including constraints on the contact foot joint points and a time-dependent inertia tensor-based constraint. The motion model refinement system obtains the refined motion model by solving the optimization problem.

The present disclosure is directed toward systems and methods that enable simultaneous viewing and editing of audio-visual content within a virtual-reality environment (i.e., while wearing a virtual-reality device). For example, the virtual-reality editing system allows for editing of audio-visual content while viewing the audio-visual content via a virtual-reality device. In particular, the virtual-reality editing system provides an editing interface over a display of audio-visual content provided via a virtual-reality device (e.g., a virtual-reality headset) that allows for editing of the audio-visual content.

Facilitating editing of spherical panorama videos using a virtual-reality headset

Embodiments disclosed herein describe systems, methods, and products that train one or more neural networks and execute the trained neural network across various applications. The one or more neural networks are trained to optimize a loss function comprising a pixel-level comparison between the outputs generated by the neural networks and the ground truth dataset generated from a bubble view methodology or an explicit importance maps methodology. Each of these methodologies may be more efficient than and may closely approximate the more expensive but accurate human eye gaze measurements. The embodiments herein leverage an existing process for training neural networks to generate importance maps of a plurality of graphic objects to offer interactive applications for graphics designs and data visualizations. Based on the importance maps, the computer may provide real-time design feedback, generate smart thumbnails of the graphic objects, provide recommendations for design retargeting, and extract smart color themes from the graphic objects.

Systems and methods of learning visual importance for graphic design and data visualization

Virtual reality parallax correction techniques and systems are described that are configured to correct parallax for VR digital content captured from a single point of origin. In one example, a parallax correction module is employed to correct artifacts caused in a change from a point of origin that corresponds to the VR digital content to a new viewpoint with respect to an output of the VR digital content. A variety of techniques may be employed by the parallax correction module to correct parallax. Examples of these techniques include depth filtering, boundary identification, smear detection, mesh cutting, confidence estimation, blurring, and error diffusion as further described in the following sections.

Virtual reality parallax correction

Growing community of inventors

San Francisco, CA, United States of America

Aaron Phillip Hertzmann