Snap Inc.

Himmy Tam

Riza Alp Guler

Avihay Assouline

Brian Fulkerson

Iason Kokkinos

Georgios Papandreou

Madiyar Aitbayev

Kai Zhou

Nir Malbin

Haoyang Wang

Dominik Kulon

Laura Rosalia Luidolt

Mohammad Rami Koujan

An system for augmenting images using hand surface normal estimation is provided. In a model training phase, 3D models of hands are generated using 3D data of hands in a variety of positions. Target normal training data is generated that includes normals of surfaces of the 3D models and synthetic 2D image training data corresponding to the 3D models and the normals. The target normal training data and the synthetic image training data are used to train a normal estimation model. The normal estimation is used by an interactive application to generate augmentations that are applied to hand image data.

Hand surface normal estimation

Methods and systems are disclosed for transferring garments from one real-world object to another in real time using body landmarks. The system receives a first image that includes a depiction of a first person wearing a fashion item in a first pose. The system obtains a second image that includes a depiction of a second person in a second pose and generates a first set of body landmarks corresponding the first person in the first pose and a second set of body landmarks corresponding the second person wearing in the first pose. The system computes a deviation between the first set of body landmarks and the second set of body landmarks. The system generates a new image that depicts the second person wearing the fashion item worn by the first person based on the deviation between the first set of body landmarks and the second set of body landmarks.

Real-time try-on using body landmarks

Methods and systems are disclosed for transferring garments from a real-world object to a virtual object. The system receives, by a client device, an image that includes a depiction of a real-world object having a fashion item in a real-world environment. The system accesses a three-dimensional (3D) avatar model of a human and generates a graphic item corresponding to the fashion item being worn by the real-world object depicted in the image. The system modifies the 3D avatar model of the human based on the graphic item and presents the 3D avatar model that has been modified based on the graphic item within a view of the real-world environment on the client device.

Real-time fashion item transfer system

Methods and systems are disclosed for performing operations for applying augmented reality elements to a person depicted in an image. The operations include receiving an image that includes data representing a depiction of a person; extracting a portion of the image; applying a first machine learning model stage to the portion to predict a depth of a point of interest for the data representing the depiction of the person; applying a second machine learning model stage to the portion of the image to predict a relative depth of each pixel in the portion of the image to the predicted depth of the point of interest; generating dense depth reconstruction of the data representing the depiction of the person based on outputs of the first and second stages of the machine learning model; and applying one or more AR elements to the image based on the dense depth reconstruction.

Pixel depth determination for object

Methods and systems are disclosed for performing operations for applying augmented reality elements to a person depicted in an image. The operations include receiving an image that includes data representing a depiction of a person; generating a segmentation of the data representing the person depicted in the image; extracting a portion of the image corresponding to the segmentation of the data representing the person depicted in the image; applying a machine learning model to the portion of the image to predict a surface normal tensor for the data representing the depiction of the person, the surface normal tensor representing surface normals of each pixel within the portion of the image; and applying one or more augmented reality (AR) elements to the image based on the surface normal tensor.

Growing community of inventors

London, United Kingdom

Himmy Tam