Snap Inc.

Jian Ren

Sergey Tulyakov

Menglei Chai

Kyle Olszewski

Hsin-Ying Lee

Aliaksandr Siarohin

Pavel Savchenkov

Arnab Ghosh

Oleksandr Pyshchenko

Zeng Huang

Qing Jin

Ju Hu

Huan Wang

Ning Xu

Jianchao Yang

Jiayao Yu

Oliver Woodford

Sergey Smetanin

Artem Bondich

Richard Leider

Ivan Skorokhodov

Kavya Venkata Kota Kopparapu

Benjamin Dodson

Francesc Xavier Drudis Rius

Angus Kong

Yanyu Li

Daniel Moreno

Dhritiman Sagar

Aleksei Stoliar

Georgios Evangelidis

Riza Alp Guler

Roman Golobokov

Yang Wen

Nikita Demidov

Ivan Babanin

Nikita Belosludtcev

Dor Ayalon

Denys Makoviichuk

Timur Zakirov

Vladimir Gordienko

Ligong Han

Vladislav Shakhrai

Willi Menapace

Aleksandr Zakharov

Yinghao Xu

Sofya Savinova

Francesco Barbieri

Panagiotis Achlioptas

Pavlo Chemerys

Junli Cao

Yash Mukund Kant

Geng Yuan

Mykyta Bakunov

Yuwei Fang

Ergeta Muca

Colin Eles

Zezhou Cheng

Tsai-Shien Chen

Shervin Minaee

Examples disclosed herein describe techniques for automatic image quality evaluation. A first set of images generated by a first automated image generator and a second set of images generated by a second automated image generator are accessed. A first machine learning model generates a first quality indicator for each image in the first set of images and the second set of images. A second machine learning model generates a second quality indicator for each image in the first set of images and the second set of images. Based on the generated indicators, a first image from the first set of images and a second image from the second set of images are automatically selected and compared. A first ranking of the first automated image generator and the second automated image generator is generated based on the comparison, and ranking data is caused to be presented on a device.

Automatic image quality evaluation

Automatic captioning pipelines and methods for automatically annotating video data with subtitles, which can be obtained using automatic speech recognition (ASR). An automatic captioning pipeline with inputs of multimodal data scales up the dataset of high-quality video-caption pairs. The automatic captioning pipeline generates video-caption pairs by establishing and using a large video-language dataset along with an automatic captioning approach leveraging multimodal inputs, such as textual video description, subtitles, and individual video frames.

Captioning videos with multiple cross-modality teachers

A neural light field (NeLF) that runs real-time on mobile devices for neural rendering of three dimensional (3D) scenes, referred to as MobileR2L. The MobileR2L architecture runs efficiently on mobile devices with low latency and small size, and it achieves high-resolution generation while maintaining real-time inference for both synthetic and real-world 3D scenes on mobile devices. The MobileR2L has a network backbone including a convolutional layer embedding an input image at a resolution, residual blocks uploading the embedded image, and super-resolution modules receiving the uploaded embedded image and rendering an output image having a higher resolution than the embedded image. The convolution layer generates a number of rays equal to a number of pixels in the input image, where a partial number of the rays is uploaded to the super-resolution modules.

Real-time neural light field on mobile devices

Described is a system for improving machine learning models. In some cases, the system improves such models by identifying a performance characteristic for machine learning model blocks in an iterative denoising process of a machine learning model, connecting a prior machine learning model block with a subsequent machine learning model block of the machine learning model blocks within the machine learning model based on the identified performance characteristic, identifying a prompt of a user, the prompt indicative of an intent of the user for generative images, and analyzing data corresponding to the prompt using the machine learning model to generate one or more images, the machine learning model trained to generate images based on data corresponding to prompts.

Text-to-image diffusion model rearchitecture

Methods and systems are disclosed for performing operations for generating a 3D model of a scene. The operations include: receiving a set of two-dimensional (2D) images representing a first view of a real-world environment; applying a machine learning model comprising a neural light field network to the set of 2D images to predict pixel values of a target image representing a second view of the real-world environment, the machine learning model being trained to map a ray origin and direction directly to a given pixel value; and generating a three-dimensional (3D) model of the real-world environment based on the set of 2D images and the predicted target image.

3D modeling based on neural light field

A method of generating an image for use in a conversation taking place in a messaging application is disclosed. Conversation input text is received from a user of a portable device that includes a display. Model input text is generated from the conversation input text, which is processed with a text-to-image model to generate an image based on the model input text. The coordinates of a face in the image are determined, and the face of the user or another person is added to the image at the location. The final image is displayed on the portable device, and user input is received to transmit the image to a remote recipient.

Text-guided cameo generation

Invertible Neural Networks (INNs) are used to build an Invertible Neural Skinning (INS) pipeline for reposing characters during animation. A Pose-conditioned Invertible Network (PIN) is built to learn pose-conditioned deformations. The end-to-end Invertible Neural Skinning (INS) pipeline is produced by placing two PINs around a differentiable Linear Blend Skinning (LBS) module using a pose-free canonical representation. The PINs help capture the non-linear surface deformations of clothes across poses and alleviate the volume loss suffered from the LBS operation. Since the canonical representation remains pose-free, the expensive mesh extraction is performed exactly once, and the mesh is reposed by warping it with the learned LBS during an inverse pass through the INS pipeline.

Invertible neural skinning

Unsupervised volumetric 3D animation (UVA) of non-rigid deformable objects without annotations learns the 3D structure and dynamics of objects solely from single-view red/green/blue (RGB) videos and decomposes the single-view RGB videos into semantically meaningful parts that can be tracked and animated. Using a 3D autodecoder framework, paired with a keypoint estimator via a differentiable perspective-n-point (PnP) algorithm, the UVA model learns the underlying object 3D geometry and parts decomposition in an entirely unsupervised manner from still or video images. This allows the UVA model to perform 3D segmentation, 3D keypoint estimation, novel view synthesis, and animation. The UVA model can obtain animatable 3D objects from a single or a few images. The UVA method also features a space in which all objects are represented in their canonical, animation-ready form. Applications include the creation of lenses from images or videos for social media applications.

Unsupervised volumetric animation

A three-dimensional (3D) scene is generated from non-aligned generic camera priors by producing a tri-plane representation for an input scene received in random latent code, obtaining a camera posterior including posterior parameters representing color and density data from the random latent code and from generic camera priors without alignment assumptions, and volumetrically rendering an image of the input scene from the color and density data to provide a scene having pixel colors and depth values from an arbitrary camera viewpoint. A depth adaptor processes depth values to generate an adapted depth map that bridges domains of rendered and estimated depth maps for the image of the input scene. The adapted depth map, color data, and scene geometry information from an external dataset are provided to a discriminator for selection of a 3D representation of the input scene.

3D generation of diverse categories and scenes

A multimodal video generation framework (MMVID) that benefits from text and images provided jointly or separately as input. Quantized representations of videos are utilized with a bidirectional transformer with multiple modalities as inputs to predict a discrete video representation. A new video token trained with self-learning and an improved mask-prediction algorithm for sampling video tokens is used to improve video quality and consistency. Text augmentation is utilized to improve the robustness of the textual representation and diversity of generated videos. The framework incorporates various visual modalities, such as segmentation masks, drawings, and partially occluded images. In addition, the MMVID extracts visual information as suggested by a textual prompt.

Video synthesis via multimodal conditioning

Aspects of the present disclosure involve a system comprising a medium storing a program and method for machine-learning based selection of a representative video frame. The program and method provide for receiving a set of video frames; determining a first subset of frames by removing frames outside of an image quality threshold; determining a second subset by removing frames outside of an image stillness threshold; computing feature data for each frame in the second subset; providing, for each frame in the second subset, the feature data to a machine learning model (MLM), the MLM being configured to output a score for each frame in the second subset of frames based on the feature data, the MLM having been trained with a first set of images labeled based on aesthetics, and with a second set of images labeled based on image quality; and selecting a frame based on output scores.

Machine learning-based selection of a representative video frame within a messaging application

A system and method are described for generating 3D garments from two-dimensional (2D) scribble images drawn by users. The system includes a conditional 2D generator, a conditional 3D generator, and two intermediate media including dimension-coupling color-density pairs and flat point clouds that bridge the gap between dimensions. Given a scribble image, the 2D generator synthesizes dimension-coupling color-density pairs including the RGB projection and density map from the front and rear views of the scribble image. A density-aware sampling algorithm converts the 2D dimension-coupling color-density pairs into a 3D flat point cloud representation, where the depth information is ignored. The 3D generator predicts the depth information from the flat point cloud. Dynamic variations per garment due to deformations resulting from a wearer's pose as well as irregular wrinkles and folds may be bypassed by taking advantage of 2D generative models to bridge the dimension gap in a non-parametric way.

3D garment generation from 2D scribble images

A messaging system performs neural network hair rendering for images provided by users of the messaging system. A method of neural network hair rendering includes processing a three-dimensional (3D) model of fake hair and a first real hair image depicting a first person to generate a fake hair structure, and encoding, using a fake hair encoder neural subnetwork, the fake hair structure to generate a coded fake hair structure. The method further includes processing, using a cross-domain structure embedding neural subnetwork, the coded fake hair structure to generate a fake and real hair structure, and encoding, using an appearance encoder neural subnetwork, a second real hair image depicting a second person having a second head to generate an appearance map. The method further includes processing, using a real appearance renderer neural subnetwork, the appearance map and the fake and real hair structure to generate a synthesized real image.

Messaging system with neural hair rendering

A vision transformer network having extremely low latency and usable on mobile devices, such as smart eyewear devices and other augmented reality (AR) and virtual reality (VR) devices. The transformer network processes an input image, and the network includes a convolution stem configured to patch embed the image. A first stack of stages including at least two stages of 4-Dimension (4D) metablocks (MBs) (MB) follow the convolution stem. A second stack of stages including at least two stages of 3-Dimension MBs (MB) follow the MBstages. Each of the MBstages and each of the MBstages include different layer configurations, and each of the MBstages and each of the MBstages include a token mixer. The MBstages each additionally include a multi-head self attention (MHSA) processing block.

Efficientformer vision transformer

Examples disclosed herein describe techniques related to automated image generation in an interaction system. An image generation request is received from a first user device associated with a first user of an interaction system. The image generation request comprises a text prompt. Responsive to receiving the image generation request, an image is automatically generated by an automated text-to-image generator, based on the text prompt. The image is caused to be presented on the first user device. An indication of user input to select the image is received from the user device. Responsive to receiving the indication of the user input to select the image, the image is associated with the first user within the interaction system, and a second user of the interaction system is enabled to be presented with the image.

Automatic image generation in an interaction system

Growing community of inventors

Highland Park, NJ, United States of America

Jian Ren