Snap Inc.

Aleksei Stoliar

Sergey Tulyakov

Sergey Demyanov

Aleksei Podkin

Sergei Korolev

Victor Shaburov

Oleksandr Pyshchenko

Grygoriy Kozhemiak

Maksim Gusarov

Trevor Stephenson

Christopher Yale Crutchfield

Vadim Velicodnii

Andrew Wan

Fedor Zhdanov

Roman Furko

Walton Lin

Andrew Cheng-min Lin

Jonathan Solichin

Enxu Yan

Aliaksandr Siarohin

Sergei Kotcur

Olha Rykhliuk

Vladislav Shakhrai

Dhritiman Sagar

Jian Ren

Hsin-Ying Lee

Istvan Marton

Huan Wang

Qing Jin

Ju Hu

Yanyu Li

Mikhail Vasilkovskii

Ergeta Muca

Colin Eles

Pavlo Chemerys

Songfang Han

An artificial intelligence (AI) network or neural network is trained to generate three-dimensional (3D) models or shapes with color from two-dimensional (2D) input images and input text describing the 3D model with color. Example methods include converting a first three-dimensional (3D) model from a first representation to a second representation, the second representation including color information for the 3D model and inputting the second representation into an encoder to generate a third representation having a lower dimension than the second representation. The method further includes inputting the third representation into a decoder to generate a fourth representation having a same dimension as the second representation and generating a second 3D model from the fourth representation. The method further includes determining losses between the first 3D model and the second 3D model and updating weights of the encoder and the decoder based on the losses.

Generating 3D models with texture

Described is a system for improving machine learning models. In some cases, the system improves such models by identifying a performance characteristic for machine learning model blocks in an iterative denoising process of a machine learning model, connecting a prior machine learning model block with a subsequent machine learning model block of the machine learning model blocks within the machine learning model based on the identified performance characteristic, identifying a prompt of a user, the prompt indicative of an intent of the user for generative images, and analyzing data corresponding to the prompt using the machine learning model to generate one or more images, the machine learning model trained to generate images based on data corresponding to prompts.

Text-to-image diffusion model rearchitecture

The subject technology receives an object mesh, information related to a viewpoint for rendering an image of an object having a reflective surface, and a set of maps. The subject technology generates a rasterized RGB (Red Green Blue) image based on the object mesh, the viewpoint, and the set of maps. The subject technology generates, using a neural network model, an output image of the object with the reflective surface based at least in part on the rasterized RGB image and the viewpoint. The subject technology provides for display the output image of the object with the reflective surface on a display of a computer client device.

Neural shading of reflective surfaces

A neural network pruning system can sparsely prune neural network models using an optimizer based approach that is agnostic to the model architecture being pruned. The neural network pruning system can prune by operating on the parameter vector of the full model and the gradient vector of the loss function with respect to the model parameters. The neural network pruning system can iteratively update parameters based on the gradients, while zeroing out as many parameters as possible based a preconfigured penalty.

Optimizer based prunner for neural networks

A compact generative neural network can be distilled from a teacher generative neural network using a training network. The compact network can be trained on the input data and output data of the teacher network. The training network train the student network using a discrimination layer and one or more types of losses, such as perception loss and adversarial loss.

Generative neural network distillation

Systems and methods herein describe receiving an image via an image capture device, using a machine learning model, generating an image augmentation decision, accessing an augmented reality content item, associating the generated image augmentation decision with the augmented reality content item, modifying the received image using the augmented reality content item and the associated image augmentation decision, and causing presentation of the modified image on a graphical user interface of a computing device.

Machine learning in augmented reality content items

Systems and methods are disclosed for generating, a source image sequence using an image sensor of the computing device, the source image sequence comprising a plurality of source images depicting a head and face, identifying driving image sequence data to modify face image feature data in the source image sequence, generating, using an image transformation neural network, a modified source image sequence comprising a plurality of modified source images depicting modified versions of the head and face, and storing the modified source image sequence on the computing device.

Facial animation transfer

A landmark detection system can more accurately detect landmarks in images using a detection scheme that penalizes for dispersion parameters, such as variance or scale. The landmark detection system can be trained using both labeled and unlabeled training data in a semi-supervised approach. The landmark detection system can further implement tracking of an object across multiple images using landmark data.

Image landmark detection

Systems and methods herein describe a video compression system. The described systems and methods accesses a sequence of image frames from a first computing device, the sequence of image frames comprising a first image frame and a second image frame, detects a first set of keypoints for the first image frame, transmits the first image frame and the first set of keypoints to a second computing device, detects a second set of keypoints for the second image frame, transmits the second set of keypoints to the second computing device, causes an animated image to be displayed on the second computing device.

Video compression system

Methods and systems are disclosed for performing operations for generating a photorealistic rendering of an object. The operations include: accessing a set of albedo textures and a machine learning model associated with a real-world object, the set of albedo textures and a machine learning model having been trained based on a plurality of viewpoints of the real-world object; obtaining a three-dimensional (3D) mesh of the real-world object; receiving input that selects a new viewpoint that differs from the plurality of viewpoints of the real-world object; and generating a photorealistic rendering of the real-world object from the new viewpoint based on the 3D mesh of the real-world object, the set of albedo textures, and the machine learning model associated with the real-world object.

Neural rendering using trained albedo textures

A data item is identified on a device. A neural network that includes an adversarial transformation subnetwork is applied to the data item to generate a modified data item. Output indicative of the modified data item is caused to be presented on the device. The neural network further comprises an encoder and a decoder. The neural network is trained in at least two stages. At least one of the encoder and the decoder is trained in a first stage and the adversarial transformation subnetwork is trained in a second stage.

Deep feature generative adversarial neural networks

A mobile device can implement a neural network-based domain transfer scheme to modify an image in a first domain appearance to a second domain appearance. The domain transfer scheme can be configured to detect an object in the image, apply an effect to the image, and blend the image using color space adjustments and blending schemes to generate a realistic result image. The domain transfer scheme can further be configured to efficiently execute on the constrained device by removing operational layers based on resources available on the mobile device.

Growing community of inventors

Marina Del Rey, CA, United States of America

Aleksei Stoliar