Snap Inc.

Haoyang Wang

Riza Alp Guler

Matan Zohar

Georgios Papandreou

Brian Fulkerson

Yanli Zhao

Frank Lu

Itamar Berger

Avihay Assouline

Iason Kokkinos

Stefanos Zafeiriou

Himmy Tam

Dominik Kulon

Antonios Kakolyris

An system for augmenting images using hand surface normal estimation is provided. In a model training phase, 3D models of hands are generated using 3D data of hands in a variety of positions. Target normal training data is generated that includes normals of surfaces of the 3D models and synthetic 2D image training data corresponding to the 3D models and the normals. The target normal training data and the synthetic image training data are used to train a normal estimation model. The normal estimation is used by an interactive application to generate augmentations that are applied to hand image data.

Hand surface normal estimation

Methods and systems are disclosed for generating a body mesh from a single image. The system predicts both a volumetric reconstruction tensor of the monocular image and a pose of an object by applying a first machine learning model to a monocular image. The system identifies a portion of the pose of the object that corresponds to a point in a canonical space associated with a set of position encoding information. The system obtains a point of the volumetric reconstruction tensor corresponding to the identified portion of the pose. The system classifies the obtained point as being inside or outside of a canonical volume by applying a second machine learning model to the obtained point of the volumetric reconstruction tensor together with the set of position encoding information. The system generates a three-dimensional (3D) mesh representing the object in the canonical space.

Body mesh reconstruction from RGB image

Methods and systems are disclosed for performing operations comprising: receiving a video that includes a depiction of a real-world object; generating a three-dimensional (3D) body mesh associated with the real-world object that tracks movement of the real-world object across frames of the video; obtaining an external mesh associated with an augmented reality element; automatically establishing a correspondence between the 3D body mesh associated with the real-world object and the external mesh; deforming the external mesh based on movement of the real-world object and the established correspondence with the 3D body mesh; and modifying the video to include a display of the augmented reality element based on the deformed external mesh.

Deforming custom mesh based on body mesh

Aspects of the present disclosure involve a system for providing virtual experiences. The system accesses, by a messaging application, an image depicting a person. The system generates, by the messaging application, a three-dimensional (3D) avatar based on the person depicted in the image. The system receives input that selects a pose for the 3D avatar and one or more fashion items to be worn by the 3D avatar and places, by the messaging application, the 3D avatar in the selected pose and wearing the one or more fashion items in an augmented reality (AR) experience.

Applying animated 3D avatar in AR experiences

Aspects of the present disclosure involve a system and a method for performing operations comprising: receiving a plurality of bone scale coefficients each corresponding to respective bones of a skeleton model; receiving a plurality of joint angle coefficients that collectively define a pose for the skeleton model; generating the skeleton model based on the received bone scale coefficients and the received joint angle coefficients; generating a base surface based on the plurality of bone scale coefficients; generating an identity surface by deformation of the base surface; and generating the 3D body model by mapping the identity surface onto the posed skeleton model.

3D body model generation

Growing community of inventors

London, United Kingdom

Haoyang Wang