Max-planck-gesellschaft Zur F&#xf6;rderung Der Wissenschaften E. V.

Amazon Technologies, Inc.

Lucasfilm Entertainment Company Ltd.

Matthew Maverick Loper

Michael Julian Black

Nicolas Heron

Eric Rachlin

Alexander Weiss

David Allen Smith

Javier Romero Gonzalez-Nicolas

Naureen Mahmood

Javier Romero

Xiaochen Hu

Gerard Pons-Moll

Kiran Bhat

Steven Douglas Hatch

Stéphane Grabli

William John O'Farrell

Jason Lawrence Gelman

Theodora Hinkle

Martin Kristiansen

Evan Lee

Jonathan Robert Cilley

Disclosed are systems and techniques for extracting user body shape (e.g., a representation of the three-dimensional body surface) from user behavioral data. The behavioral data may not be explicitly body-shape-related, and can include shopping history, social media likes, or other recorded behaviors of the user within (or outside of) a networked content delivery environment. The determined body shape can be used, for example, to generate a virtual fitting room user interface.

Machine learning systems and methods for extracting user body shape from behavioral data

A system configured to perform an accurate and fast estimation of an object shape from a single input image. The system may process image data representing a first surface of an object using image-to-image translation techniques. A first trained model may generate depth information for the object, such as front distance estimates and back distance estimates. The system may use the depth information to generate an output mesh shaped like the object, such as, in the case of a pliable object a reposable avatar. The system may improve depth estimation by including a loss on surface normals in the first trained model. A second trained model may generate color information to be applied to the output mesh to accurately represent the object. The output mesh may include detailed geometry and appearance of the object, useful for a variety of purposes such as gaming, virtual/augmented reality, virtual shopping, and other implementations.

Detailed 3D estimation from a single image

A method of transferring a facial expression from a subject to a computer generated character that includes: receiving a plate with an image of the subject's facial expression and an estimate of intrinsic parameters of a camera used to film the plate; generating a three-dimensional parameterized deformable model of the subject's face where different facial expressions of the subject can be obtained by varying values of the model parameters; solving for the facial expression in the plate by executing a deformation solver to solve for at least some parameters of the deformable model with a differentiable renderer and shape-from-shading techniques, using as inputs, the three-dimensional parameterized deformable model, estimated intrinsic camera parameters, estimated lighting conditions and albedo estimates over a series of iterations to infer geometry of the facial expression and generate an intermediate facial; generating, from the intermediate facial mesh, refined albedo estimates for the deformable model; and solving for the facial expression in the plate by executing the deformation solver using the intermediate facial mesh, the estimated intrinsic camera parameters, the estimated lighting conditions and the refined albedo estimates as inputs over a series of iterations to infer geometry of the facial expression and generate a final facial mesh using the set of parameter values of the deformable model which result in a facial expression that more closely matches the expression of the subject in the plate than does the intermediate facial mesh.

Facial performance capture in an uncontrolled environment

The invention comprises a learned model of human body shape and pose dependent shape variation that is more accurate than previous models and is compatible with existing graphics pipelines. Our Skinned Multi-Person Linear model (SMPL) is a skinned vertex based model that accurately represents a wide variety of body shapes in natural human poses. The parameters of the model are learned from data including the rest pose template, blend weights, pose-dependent blend shapes, identity-dependent blend shapes, and a regressor from vertices to joint locations. Unlike previous models, the pose-dependent blend shapes are a linear function of the elements of the pose rotation matrices. This simple formulation enables training the entire model from a relatively large number of aligned 3D meshes of different people in different poses. The invention quantitatively evaluates variants of SMPL using linear or dual quaternion blend skinning and show that both are more accurate than a BlendSCAPE model trained on the same data. In a further embodiment, the invention realistically models dynamic soft-tissue deformations. Because it is based on blend skinning, SMPL is compatible with existing rendering engines and we make it available for research purposes.

Skinned multi-person linear model

Disclosed is a method including receiving an input image including a human, predicting, based on a convolutional neural network that is trained using examples consisting of pairs of sensor data, a corresponding body shape of the human and utilizing the corresponding body shape predicted from the convolutional neural network as input to another convolutional neural network to predict additional body shape metrics.

Machine learning systems and methods of estimating body shape from images

Disclosed is a method including receiving visual input comprising a human within a scene, detecting a pose associated with the human using a trained machine learning model that detects human poses to yield a first output, estimating a shape (and optionally a motion) associated with the human using a trained machine learning model associated that detects shape (and optionally motion) to yield a second output, recognizing the scene associated with the visual input using a trained convolutional neural network which determines information about the human and other objects in the scene to yield a third output, and augmenting reality within the scene by leveraging one or more of the first output, the second output, and the third output to place 2D and/or 3D graphics in the scene.

Machine learning systems and methods for augmenting images

The invention comprises a learned model of human body shape and pose dependent shape variation that is more accurate than previous models and is compatible with existing graphics pipelines. Our Skinned Multi-Person Linear model (SMPL) is a skinned vertex based model that accurately represents a wide variety of body shapes in natural human poses. The parameters of the model are learned from data including the rest pose template, blend weights, pose-dependent blend shapes, identity-dependent blend shapes, and a regressor from vertices to joint locations. Unlike previous models, the pose-dependent blend shapes are a linear function of the elements of the pose rotation matrices. This simple formulation enables training the entire model from a relatively large number of aligned 3D meshes of different people in different poses. The invention quantitatively evaluates variants of SMPL using linear or dual-quaternion blend skinning and show that both are more accurate than a Blend SCAPE model trained on the same data. In a further embodiment, the invention realistically models dynamic soft-tissue deformations. Because it is based on blend skinning, SMPL is compatible with existing rendering engines and we make it available for research purposes.

Growing community of inventors

San Francisco, CA, United States of America

Matthew Maverick Loper