Disney Enterprises, Inc.

Carnegie Mellon University

Two Pic Mc LLC

Jessica Kate Hodgins

Katsu Yamane

Leonid Sigal

Ariel Shamir

Yaser Sheikh

Moshe Benjamin Mahler

Yuka Ariki

Lavanya Sharan

Matthias Grundmann

Iain Matthews

Irfan Essa

Takaaki Shiratori

Yu Zheng

Michael Mistry

Adrien Treuille

Ronit Slyper

Binh H Le

Marek Vondrak

Libin Liu

Peng Guan

Akihiko Murai

Alizreza Fathi

Edilson De Aguiar

James Rehg

Kihwan Kim

Eakta Jain

Hyun Soo Park

Sang Il Park

Edilson DeAguiar

The disclosure provides an approach for learning to schedule control fragments for physics-based virtual character simulations and physical robot control. Given precomputed tracking controllers, a simulation application segments the controllers into control fragments and learns a scheduler that selects control fragments at runtime to accomplish a task. In one embodiment, each scheduler may be modeled with a Q-network that maps a high-level representation of the state of the simulation to a control fragment for execution. In such a case, the deep Q-learning algorithm applied to learn the Q-network schedulers may be adapted to use a reward function that prefers the original controller sequence and an exploration strategy that gives more chance to in-sequence control fragments than to out-of-sequence control fragments. Such a modified Q-learning algorithm learns schedulers that are capable of following the original controller sequence most of the time while selecting out-of-sequence control fragments when necessary.

Learning to schedule control fragments for physics-based character simulation and robots using deep Q-learning

Systems and methods of virtual entity animation are presented herein. A virtual entity may have virtual skin defined by a model having a set of vertices. A skeletal-based skin deformation method of animation may be used wherein individual vertices in the model may have individually and independently determined centers of rotation.

Systems and methods for virtual entity animation

Techniques are disclosed for providing a learning-based clothing model that enables the simultaneous animation of multiple detailed garments in real-time. A simple conditional model learns and preserves key dynamic properties of cloth motions and folding details. Such a conditional model may be generated for each garment worn by a given character. Once generated, the conditional model may be used to determine complex body/cloth interactions in order to render the character and garment from frame-to-frame. The clothing model may be used for a variety of garments worn by male and female human characters (as well as non-human characters) while performing a varied set of motions typically used in video games (e.g., walking, running, jumping, turning, etc.).

Stable spaces for rendering character garments in real-time

Described herein are methods, systems, apparatuses and products for utilizing motion fields to predict evolution in dynamic scenes. One aspect provides for accessing active object position data including positioning information of a plurality of individual active objects; extracting a plurality of individual active object motions from the active object position data; constructing a motion field using the plurality of individual active object motions; and using the motion field to predict one or more points of convergence at one or more spatial locations that active objects are proceeding towards at a future point in time. Other embodiments are disclosed.

System and method for utilizing motion fields to predict evolution in dynamic scenes

Various embodiments of the invention provide a control framework for robots such that a robot can use all joints simultaneously to track motion capture data and maintain balance. Embodiments of the invention provide a framework enabling complex reference movements to be automatically tracked, for example reference movements derived from a motion capture data system.

Systems and methods for tracking and balancing robots for imitating motion capture data

One embodiment of the present invention sets forth a technique for generating a three-dimensional (3D) model of an individual. The technique involves receiving pose data that includes image data associated with the individual being in a pose, selecting from a database a morphable 3D generic human model based on a gender of the individual, adjusting a pose of the morphable 3D generic model based on the image data, identifying, based on the image data, one or more 3D appearance parameter objects included in the database, and adjusting via one or more processors one or more properties of the morphable 3D generic model based on the image data as well as the one or more 3D appearance parameter objects to produce the 3D model of the individual.

Perceptually guided capture and stylization of 3D human figures

The disclosure provides an approach for estimating a state-space controller from a set of video frames depicting a motion of an entity. The approach includes incrementally optimizing parameters of the state-space controller and changing a structure of the state-space controller based on expanding subsets of the set of video frames. In one embodiment, a controller-estimation application greedily selects, at every stage of the incremental optimization, structure and parameters of the controller which minimize an objective function. In another embodiment, the controller-estimation application re-optimizes, after the incremental optimization, all parameters of the state-space controller based on all of the video frames. In yet a further embodiment, the controller-estimation application alters the structure of the state-space controller for robustness and compactness by adding cycles in the state-space controller and enforcing constraints on the structure of the state-space controller and adding and modifying state transition types, as appropriate.

Video-based motion capture and adaptation

Systems and method for modeling hair in real-time with user interactive controls are presented. One embodiment may take the form of a method of hair motion modeling including representing hair with hair guides, each hair guide comprising a plurality of hair points and reducing a dimensionality of the hair guides to achieve a reduced sub-space. Additionally, the method includes generating a data tensor for multiple factors related to the hair guides and decomposing the tensor to create a model characterizing the multiple factors in a multi-linear hair framework. The hair may be hair, such as human hair, animal fur, or clothing fibers.

Multi-linear dynamic hair or clothing model with efficient collision handling

Embodiments of the invention provide an approach for reproducing a human action with a robot. The approach includes receiving data representing motions and contact forces of the human as the human performs the action. The approach further includes approximating, based on the motions and contact forces data, the center of mass (CoM) trajectory of the human in performing the action. Finally, the approach includes generating a planned robot action for emulating the designated action by solving an inverse kinematics problem having the approximated human CoM trajectory as a hard constraint and the motion capture data as a soft constraint.

Robot action based on human demonstration

A system and method for providing a plurality of frames from a video, the video having been taken from a first person's perspective, identifying patterns of attention depicted by living beings appearing in the plurality of frames, identifying social interactions associated with the plurality of frames using the identified patterns of attention over a period of time and using the identified social interactions to affect the subsequent use, storage, presentation, or processing of the plurality of frames.

Method and system for detecting and recognizing social interactions in a video

Techniques are disclosed for optimizing and maintaining cyclic biped locomotion of a robot on an object. The approach includes simulating trajectories of the robot in contact with the object. During each trajectory, the robot maintains balance on the object, while using the object for locomotion. The approach further includes determining, based on the simulated trajectories, an initial state of a cyclic gait of the robot such that the simulated trajectory of the robot starting from the initial state substantially returns to the initial state at an end of one cycle of the cyclic gait. In addition, the approach includes sending joint angles and joint velocities of the initial state to a set of joint controllers of the robot to cause a leg of the robot to achieve the initial state so the robot moves through one or more cycles of the cyclic gait.

Robot cyclic locomotion using a dynamic object

Systems, methods and products for animating non-humanoid characters with human motion are described. One aspect includes selecting key poses included in initial motion data at a computing system; obtaining non-humanoid character key poses which provide a one to one correspondence to selected key poses in said initial motion data; and statically mapping poses of said initial motion data to non-humanoid character poses using a model built based on said one to one correspondence from said key poses of said initial motion data to said non-humanoid character key poses. Other embodiments are described.

Systems and methods for animating non-humanoid characters with human motion data

Techniques are disclosed for augmenting hand-drawn animation of human characters with three-dimensional (3D) physical effects to create secondary motion. Secondary motion, or the motion of objects in response to that of the primary character, is widely used to amplify the audience's response to the character's motion and to provide a connection to the environment. These 3D effects are largely passive and tend to be time consuming to animate by hand, yet most are very effectively simulated in current animation software. The techniques enable hand-drawn characters to interact with simulated objects such as cloth and clothing, balls and particles, and fluids. The driving points or volumes for the secondary motion are tracked in two dimensions, reconstructed into three dimensions, and used to drive and collide with the simulated objects.

System and method for augmenting hand animation with three-dimensional secondary motion

There is provided a system and method for database driven action capture. By utilizing low cost, lightweight MEMS devices such as accelerometers, a user friendly, wearable, and cost effective system for motion capture is provided, which relies on a motion database of previously recorded motions to reconstruct the actions of a user. By relying on the motion database, calculation errors such as integration drift are avoided and the need for complex and expensive positional compensation hardware is avoided. The accelerometers may be implemented in an E-textile embodiment using inexpensive off-the-shelf components. In some embodiments, compression techniques may be used to accelerate linear best match searching against the motion database. Adjacent selected motions may also be blended together for improved reconstruction results and visual rendering quality. Various perceivable effects may be triggered in response to the reconstructed motion, such as animating a 3D avatar, playing sounds, or operating a motor.

System and method for database driven action capture

Body-mounted cameras are used to accurately reconstruct the motion of a subject. Outward-looking cameras are attached to the limbs of the subject, and the joint angles and root pose that define the subject's configuration are estimated through a non-linear optimization, which can incorporate image matching error and temporal continuity of motion. Instrumentation of the environment is not required, allowing for motion capture over extended areas and in outdoor settings.

Growing community of inventors

Pittsburgh, PA, United States of America

Jessica Kate Hodgins