Disney Enterprises, Inc.

Eth Zurich (eidgenossische Technische Hochschule Zurich)

Eth Zurich

Eth Zurich (eidgenoessische Technische Hochschule Zurich)

Martin Guay

Robert Walker Sumner

Dominik Tobias Borer

Jakob Joachim Buhmann

Mattia Ryffel

Christoph Maurhofer

Gökcen Cimen

Stelian Coros

Ye Yuan

Ahmet Cengiz Öztireli

Maurizio Nitti

Moritz Geilinger

Gökçen Çimen

Simone Guggiari

An image processing system includes a computing platform having a hardware processor and a system memory storing an image augmentation software code, a three-dimensional (3D) shapes library, and/or a 3D poses library. The image processing system also includes a two-dimensional (2D) pose estimation module communicatively coupled to the image augmentation software code. The hardware processor executes the image augmentation software code to provide an image to the 2D pose estimation module and to receive a 2D pose data generated by the 2D pose estimation module based on the image. The image augmentation software code identifies a 3D shape and/or a 3D pose corresponding to the image using an optimization algorithm applied to the 2D pose data and one or both of the 3D poses library and the 3D shapes library, and may output the 3D shape and/or 3D pose to render an augmented image on a display.

Automated image augmentation using a virtual character

Various embodiments disclosed herein provide techniques for generating image data of a three-dimensional (3D) animatable asset. A rendering module executing on a computer system accesses a machine learning model that has been trained via first image data of the 3D animatable asset generated from first rig vector data. The rendering module receives second rig vector data. The rendering module generates, via the machine learning model, a second image data of the 3D animatable asset based on the second rig vector data.

Rig-space neural rendering of digital assets

Techniques are disclosed for learning a machine learning model that maps control data, such as renderings of skeletons, and associated three-dimensional (3D) information to two-dimensional (2D) renderings of a character. The machine learning model may be an adaptation of the U-Net architecture that accounts for 3D information and is trained using a perceptual loss between images generated by the machine learning model and ground truth images. Once trained, the machine learning model may be used to animate a character, such as in the context of previsualization or a video game, based on control of associated control points.

Techniques for feature-based neural rendering

In various embodiments, a training application generates training items for three-dimensional (3D) pose estimation. The training application generates multiple posed 3D models based on multiple 3D poses and a 3D model of a person wearing a costume that is associated with multiple visual attributes. For each posed 3D model, the training application performs rendering operation(s) to generate synthetic image(s). For each synthetic image, the training application generates a training item based on the synthetic image and the 3D pose associated with the posed 3D model from which the synthetic image was rendered. The synthetic images are included in a synthetic training dataset that is tailored for training a machine-learning model to compute estimated 3D poses of persons from two-dimensional (2D) input images. Advantageously, the synthetic training dataset can be used to train the machine-learning model to accurately infer the orientations of persons across a wide range of environments.

Techniques for inferring three-dimensional poses from two-dimensional images

Techniques are disclosed for estimating poses from images. In one embodiment, a machine learning model, referred to herein as the 'detector,' is trained to estimate animal poses from images in a bottom-up fashion. In particular, the detector may be trained using rendered images depicting animal body parts scattered over realistic backgrounds, as opposed to renderings of full animal bodies. In order to make appearances of the rendered body parts more realistic so that the detector can be trained to estimate poses from images of real animals, the body parts may be rendered using textures that are determined from a translation of rendered images of the animal into corresponding images with more realistic textures via adversarial learning. Three-dimensional poses may also be inferred from estimated joint locations using, e.g., inverse kinematics.

Joint estimation from images

An automated costume augmentation system includes a computing platform having a hardware processor and a system memory storing a software code. The hardware processor executes the software code to provide an image including a posed figure to an artificial neural network (ANN), receive from the ANN a 2D skeleton data including joint positions corresponding to the posed figure, and determine a 3D pose corresponding to the posed figure using an optimization algorithm applied to the skeleton data. The software code further identifies one or more proportion(s) of the posed figure based on the skeleton data, determines bone directions corresponding to the posed figure using another optimization algorithm applied to the 3D pose, parameterizes a costume for the posed figure based on the 3D pose, the proportion(s), and the bone directions, and outputs an enhanced image including the posed figure augmented with the fitted costume for rendering on a display.

Automated costume augmentation using shape estimation

Techniques for generating locomotion data for animating a virtual quadruped model, starting at an origin point and travelling to a destination point along a defined path. A virtual skeletal structure for the virtual quadruped model is analyzed to identify a torso region, limbs each ending in a respective end effector, and limb attributes. A predefined locomotion template for virtual quadruped characters is retrieved and mapped to the virtual quadruped model by aligning the torso region and the plurality of limbs of the virtual skeletal structure for the virtual quadruped with a second torso region and a second plurality of limbs of the predefined locomotion template. Locomotion data is generated for the virtual quadruped model based on the defined path and by upscaling the mapped predefined locomotion template, based at least on the set of limb attributes determined by analyzing the virtual skeletal structure for the virtual quadruped model.

Automatically generating quadruped locomotion controllers

A simulation engine models interactions between a simulated character and real-world objects to produce a physically realistic augmented reality (AR) simulation. The simulation engine recognizes a given real-world object and then identifies, within a library of object models, an object model corresponding to that object. The simulation engine projects the object model onto the real-world object such that the object model is geometrically aligned with the real-world object. When the simulated character encounters the real-world object, the simulation engine models interactions between the simulated character and the real-world object by adjusting the kinematics of the simulated character relative to the object model associated with the real-world object.

Growing community of inventors

Zurich, Switzerland

Martin Guay