Snap Inc.

Sergei Korolev

Aleksei Stoliar

Sergey Tulyakov

Gal Dudovitch

Ma'ayan Mishin Shuvi

Peleg Harel

Maksim Gusarov

Christopher Yale Crutchfield

Andrew Wan

Chia-Hao Hsieh

Michael David Marr

Matthew Loukas Hanover

James Powderly

Daniel Colascione

Sergei Kotcur

Scott Myers

Hsin-Ying Lee

Ihor Tymchyshyn

Andrii Didkivskyi

Maksym Bekuzarov

Songfang Han

Ma'ayan Shuvi

An artificial intelligence (AI) network or neural network is trained to generate three-dimensional (3D) models or shapes with color from two-dimensional (2D) input images and input text describing the 3D model with color. Example methods include converting a first three-dimensional (3D) model from a first representation to a second representation, the second representation including color information for the 3D model and inputting the second representation into an encoder to generate a third representation having a lower dimension than the second representation. The method further includes inputting the third representation into a decoder to generate a fourth representation having a same dimension as the second representation and generating a second 3D model from the fourth representation. The method further includes determining losses between the first 3D model and the second 3D model and updating weights of the encoder and the decoder based on the losses.

Generating 3D models with texture

An extended Reality (XR) system provides methodologies for displaying virtual objects in a hand-centric XR experience. The XR system provides an XR user interface of an XR system to a user. The XR system captures video frame data of a hand of the user and detects the hand of the user based on the video frame data and a hand-detecting model. The XR system generates a cropping boundary box based on the detection of the hand and the video frame data and generates cropped video frame data based on the cropping boundary box and the video frame data. The XR system generates a 3D model of a portion of the hand of the user based on the cropped video frame data and a virtual object based on the 3D model of the portion of the hand of the user and a 3D texture. The XR displays the virtual object in the XR user interface.

Fingernail segmentation and tracking

Methods and systems are disclosed for performing operations comprising: receiving a monocular image that includes a depiction of a whole body of a user; generating a segmentation of the whole body of the user based on the monocular image; accessing a video feed comprising a plurality of monocular images received prior to the monocular image; smoothing, using the video feed, the segmentation of the whole body generated based on the monocular image to provide a smoothed segmentation; and applying one or more visual effects to the monocular image based on the smoothed segmentation.

Whole body segmentation

A compact generative neural network can be distilled from a teacher generative neural network using a training network. The compact network can be trained on the input data and output data of the teacher network. The training network train the student network using a discrimination layer and one or more types of losses, such as perception loss and adversarial loss.

Generative neural network distillation

A gesture-based wake process for an AR system is described herein. The AR system places a hand-tracking input pipeline of the AR system in a suspended mode. A camera component of the hand-tracking input pipeline detects a possible visual wake command being made by a user of the AR system. On the basis of detecting the possible visual wake command, the AR system wakes the hand-tracking input pipeline and places the camera component in a fully operational mode. If the AR system, using the hand-tracking input pipeline, verifies the possible visual wake command as an actual wake command, the AR system initiates execution of an AR application.

Visual and audio wake commands

Methods and systems are disclosed for performing operations comprising: receiving a monocular image that includes a depiction of a whole body of a user; generating a segmentation of the whole body of the user based on the monocular image by applying one or more machine learning techniques; receiving input that selects a visualization mode; and applying one or more visual effects corresponding to the visualization mode to the monocular image based on the segmentation.

Whole body visual effects

Growing community of inventors

Marina Del Rey, CA, United States of America

Sergei Korolev