Snap Inc.

Zhejiang University

Adobe, Inc.

Menglei Chai

Sergey Tulyakov

Jian Ren

Linjie Luo

Chen Cao

Kyle Olszewski

Hsin-Ying Lee

Aliaksandr Siarohin

Oliver Woodford

Kun Zhou

Zeng Huang

Ian Anthony Wehrman

Oleksandr Pyshchenko

Shubham Vij

David LeMieux

Soumyadip Sengupta

Artem Bondich

Yanlin Weng

Lvdi Wang

Huan Wang

Ivan Skorokhodov

Willi Menapace

Qing Jin

Wen Zhang

Nathan Aaron Carr

Kalyan Krishna Sunkavalli

Sunil Sharadchandra Hadap

Riza Alp Guler

Sergey Demyanov

Pavel Savchenkov

Aleksei Podkin

Istvan Marton

Anna Kovalenko

Yunqing Hu

Daniil Ostashev

Changxi Zheng

Yinghao Xu

Aleksandr Belskikh

Zezhou Cheng

Zhengfei Kuang

Panagiotis Achlioptas

Antoine Chassang

Rameen Abdal

Peihao Zhu

Chieh Lin

Yash Mukund Kant

A 3D-aware generative model for high-quality and controllable scene synthesis uses an abstract object-level representation (i.e., 3D bounding boxes without semantic annotation) as the scene layout prior, which is simple to obtain, general to describe various scene contents, and yet informative to disentangle objects and background. An overall layout for the scene is identified and then each object is located in the layout to facilitate the scene composition process. The object-level representation serves as an intuitive user control for scene editing. Based on such a prior, the system spatially disentangles the whole scene into object-centric generative radiance fields by learning on only 2D images with global-local discrimination. Once the model is trained, users can generate and edit a scene by explicitly controlling the camera and the layout of objects' bounding boxes.

Spatially disentangled generative radiance fields for controllable 3D-aware scene synthesis

An environment synthesis framework generates virtual environments from a synthesized two-dimensional (2D) satellite map of a geographic area, a three-dimensional (3D) voxel environment, and a voxel-based neural rendering framework. In an example implementation, the synthesized 2D satellite map is generated by a map synthesis generative adversarial network (GAN) which is trained using sample city datasets. The multi-stage framework lifts the 2D map into a set of 3D octrees, generates an octree-based 3D voxel environment, and then converts it into a texturized 3D virtual environment using a neural rendering GAN and a set of pseudo ground truth images. The resulting 3D virtual environment is texturized, lifelike, editable, traversable in virtual reality (VR) and augmented reality (AR) experiences, and very large in scale.

Infinite-scale city synthesis

Methods and systems are disclosed for performing operations for generating a 3D model of a scene. The operations include: receiving a set of two-dimensional (2D) images representing a first view of a real-world environment; applying a machine learning model comprising a neural light field network to the set of 2D images to predict pixel values of a target image representing a second view of the real-world environment, the machine learning model being trained to map a ray origin and direction directly to a given pixel value; and generating a three-dimensional (3D) model of the real-world environment based on the set of 2D images and the predicted target image.

3D modeling based on neural light field

Invertible Neural Networks (INNs) are used to build an Invertible Neural Skinning (INS) pipeline for reposing characters during animation. A Pose-conditioned Invertible Network (PIN) is built to learn pose-conditioned deformations. The end-to-end Invertible Neural Skinning (INS) pipeline is produced by placing two PINs around a differentiable Linear Blend Skinning (LBS) module using a pose-free canonical representation. The PINs help capture the non-linear surface deformations of clothes across poses and alleviate the volume loss suffered from the LBS operation. Since the canonical representation remains pose-free, the expensive mesh extraction is performed exactly once, and the mesh is reposed by warping it with the learned LBS during an inverse pass through the INS pipeline.

Invertible neural skinning

Unsupervised volumetric 3D animation (UVA) of non-rigid deformable objects without annotations learns the 3D structure and dynamics of objects solely from single-view red/green/blue (RGB) videos and decomposes the single-view RGB videos into semantically meaningful parts that can be tracked and animated. Using a 3D autodecoder framework, paired with a keypoint estimator via a differentiable perspective-n-point (PnP) algorithm, the UVA model learns the underlying object 3D geometry and parts decomposition in an entirely unsupervised manner from still or video images. This allows the UVA model to perform 3D segmentation, 3D keypoint estimation, novel view synthesis, and animation. The UVA model can obtain animatable 3D objects from a single or a few images. The UVA method also features a space in which all objects are represented in their canonical, animation-ready form. Applications include the creation of lenses from images or videos for social media applications.

Unsupervised volumetric animation

A system to enable 3D hair reconstruction and rendering from a single reference image which performs a multi-stage process that utilizes both a 3D implicit representation and a 2D parametric embedding space.

Single image three-dimensional hair reconstruction

Domain adaptation frameworks for producing a 3D avatar generative adversarial network (GAN) capable of generating an avatar based on a single photographic image. The 3D avatar GAN is produced by training a target domain using an artistic dataset. Each artistic dataset includes a plurality of source images, each associated with a style type, such as caricature, cartoon, and comic. The domain adaptation framework in some implementations starts with a source domain that has been trained according to a 3D GAN and a target domain trained with a 2D GAN. The framework fine-tunes the 2D GAN by training it with the artistic datasets. The resulting 3D avatar GAN generates a 3D artistic avatar and an editing module for performing semantic and geometric edits.

Avatar generation according to artistic styles

Three-dimensional object representation and re-rendering systems and methods for producing a 3D representation of an object from 2D images including the object that enables object-centric rendering. A modular approach is used that optimizes a Neural Radiance Field (NeRF) model to estimate object geometry and refine camera parameters and, then, infer surface material properties and per-image lighting conditions that fit the 2D images.

Object-centric neural decomposition for image re-rendering

A system and method are described for generating 3D garments from two-dimensional (2D) scribble images drawn by users. The system includes a conditional 2D generator, a conditional 3D generator, and two intermediate media including dimension-coupling color-density pairs and flat point clouds that bridge the gap between dimensions. Given a scribble image, the 2D generator synthesizes dimension-coupling color-density pairs including the RGB projection and density map from the front and rear views of the scribble image. A density-aware sampling algorithm converts the 2D dimension-coupling color-density pairs into a 3D flat point cloud representation, where the depth information is ignored. The 3D generator predicts the depth information from the flat point cloud. Dynamic variations per garment due to deformations resulting from a wearer's pose as well as irregular wrinkles and folds may be bypassed by taking advantage of 2D generative models to bridge the dimension gap in a non-parametric way.

3D garment generation from 2D scribble images

A method for applying lighting conditions to a virtual object in an augmented reality (AR) device is described. In one aspect, the method includes generating, using a camera of a mobile device, an image, accessing a virtual object corresponding to an object in the image, identifying lighting parameters of the virtual object based on a machine learning model that is pre-trained with a paired dataset, the paired dataset includes synthetic source data and synthetic target data, the synthetic source data includes environment maps and 3D scans of items depicted in the environment map, the synthetic target data includes a synthetic sphere image rendered in the same environment map, applying the lighting parameters to the virtual object, and displaying, in a display of the mobile device, the shaded virtual object as a layer to the image.

Light estimation method for three-dimensional (3D) rendered objects

Aspects of the present disclosure involve a system comprising a computer-readable storage medium storing at least one program and a method for accessing a set of images depicting at least a portion of a face. A set of facial regions of the face is identified, each facial region of the set of facial regions intersecting another facial region with at least one common vertex that is a member of a set of facial vertices. For each facial region of the set of facial regions, a weight formed from a set of region coefficients is generated. Based on the set of facial regions and the weight of each facial region of the set of facial regions, the face is tracked across the set of images.

Region-based stabilized face tracking

Embodiments enable virtual hair generation. The virtual hair generation can be performed by generating a first image of a face using a GAN model, applying 3D virtual hair on the first image to generate a second image with 3D virtual hair, projecting the second image with 3D virtual hair into a GAN latent space to generate a third image with virtual hair, performing a blend of the virtual hair with the first image of the face to generate a new image with new virtual hair that corresponds to the 3D virtual hair, training a neural network that receives the second image with the 3D virtual hair and provides an output image with virtual hair, and generating using the trained neural network, a particular output image with hair based on a particular input image with 3D virtual hair.

Generating virtual hairstyle using latent space projectors

A messaging system performs neural network hair rendering for images provided by users of the messaging system. A method of neural network hair rendering includes processing a three-dimensional (3D) model of fake hair and a first real hair image depicting a first person to generate a fake hair structure, and encoding, using a fake hair encoder neural subnetwork, the fake hair structure to generate a coded fake hair structure. The method further includes processing, using a cross-domain structure embedding neural subnetwork, the coded fake hair structure to generate a fake and real hair structure, and encoding, using an appearance encoder neural subnetwork, a second real hair image depicting a second person having a second head to generate an appearance map. The method further includes processing, using a real appearance renderer neural subnetwork, the appearance map and the fake and real hair structure to generate a synthesized real image.

Messaging system with neural hair rendering

An image segmentation system to perform operations that include causing display of an image within a graphical user interface of a client device, receive a set of user inputs that identify portions of a background and foreground of the image, identify a boundary of an object depicted within the image based on the set of user inputs, crop the object from the image based on the boundary, and generate a media item based on the cropped object, wherein properties of the media object, such as a size and a shape, are based on the boundary of the object.

Image segmentation system

System and methods for compressing image-to-image models. Generative Adversarial Networks (GANs) have achieved success in generating high-fidelity images. An image compression system and method adds a novel variant to class-dependent parameters (CLADE), referred to as CLADE-Avg, which recovers the image quality without introducing extra computational cost. An extra layer of average smoothing is performed between the parameter and normalization layers. Compared to CLADE, this image compression system and method smooths abrupt boundaries, and introduces more possible values for the scaling and shift. In addition, the kernel size for the average smoothing can be selected as a hyperparameter, such as a 3×3 kernel size. This method does not introduce extra multiplications but only addition, and thus does not introduce much computational overhead, as the division can be absorbed into the parameters after training.

Compressing image-to-image models with average smoothing

Growing community of inventors

Los Angeles, CA, United States of America

Menglei Chai