Adobe, Inc.

UniversitÈ Laval

Microsoft Technology Licensing, LLC

University of California

Mitsubishi Electric Research Laboratories, Inc.

Kalyan Krishna Sunkavalli

Sunil Sharadchandra Hadap

Elya Shechtman

Zhe Lin

Xiaohui Shen

Nathan Aaron Carr

Yannick Hold-Geoffroy

Zexiang Xu

Joon-Young Lee

Emiliano Gambaretto

Jonathan Eisenmann

Zhixin Shu

Milos Hasan

Sai Bi

Zhuo Hui

Jianming Zhang

Mehmet Yumer

Yi-Hsuan Tsai

Radomir Mech

Oliver Wang

Sarah Kong

Tamy Boubekeur

He Zhang

Michal Lukac

Yilin Wang

Jean-Francois Lalonde

Chetan Nanda

Liang Shi

Fujun Luan

Pulkit Gera

Mathieu Garon

Hailin Jin

Jue Wang

Brian Lynn Price

Wojciech Matusik

Matthew David Fisher

Sohrab Amirghodsi

Jose Ignacio Echevarria Vallespi

Xin Sun

David J Kriegman

Zhili Chen

Duygu Ceylan Aksit

Vladimir Kim

Federico Perazzi

Alan Lee Erickson

I-Ming Pao

Su Chen

Michael Gharbi

Hyunghwan Byun

Kevin Karsch

Ravi Ramamoorthi

Yinglan Ma

Zhihao Xia

Seoungwug Oh

Yifan Jiang

Fabian Langguth

Frederick Mandia

Meredith Payne Stotzner

Shahrukh Athar

Jinsong Zhang

Jeong Joon Park

Hengshuang Zhao

Sing Bing Kang

Michael F Cohen

Zhaowen Wang

Linjie Luo

Hanspeter Pfister

Xin Lu

Sylvain Philippe Paris

Neel Suresh Joshi

Menglei Chai

Aliakbar Darabi

Stefano Koenig Corazza

Kevin Wampler

Niloy Jyoti Mitra

Zijun Wei

Michal Lukác

Yang Liu

Siddhant Jain

Betty Leong

Tharun Mohandoss

Szymon Marek Rusinkiewicz

Julien Philip

Yukie Takahashi

Paul Augusto Guerrero

Kartik Sethi

Xuaner Zhang

Guotong Feng

Marc-André Gardner

Siying Liu

Steve Christopher Bako

Laura Sevilla-Lara

Li-Qian Ma

Yu-Ying Yeh

Christian Gagne

Akshat Dave

Quynn Megan Le

Qiangeng Xu

Maxine Perroni-Scharf

Certain aspects and features of this disclosure relate to providing a controllable, dynamic appearance for neural 3D portraits. For example, a method involves projecting a color at points in a digital video portrait based on location, surface normal, and viewing direction for each respective point in a canonical space. The method also involves projecting, using the color, dynamic face normals for the points as changing according to an articulated head pose and facial expression in the digital video portrait. The method further involves disentangling, based on the dynamic face normals, a facial appearance in the digital video portrait into intrinsic components in the canonical space. The method additionally involves storing and/or rendering at least a portion of a head pose as a controllable, neural 3D portrait based on the digital video portrait using the intrinsic components.

Controllable dynamic appearance for neural 3D portraits

A scene modeling system receives a video including a plurality of frames corresponding to views of an object and a request to display an editable three-dimensional (3D) scene that corresponds to a particular frame of the plurality of frames. The scene modeling system applies a scene representation model to the particular frame, and includes a deformation model configured to generate, for each pixel of the particular frame based on a pose and an expression of the object, a deformation point using a 3D morphable model (3DMM) guided deformation field. The scene representation model includes a color model configured to determine, for the deformation point, color and volume density values. The scene modeling system receives a modification to one or more of the pose or the expression of the object including a modification to a location of the deformation point and renders an updated video based on the received modification.

Deformable neural radiance field for editing facial pose and facial expression in neural 3D scenes

The present disclosure relates to systems, methods, and non-transitory computer readable media for accurately, efficiently, and flexibly generating harmonized digital images utilizing a self-supervised image harmonization neural network. In particular, the disclosed systems can implement, and learn parameters for, a self-supervised image harmonization neural network to extract content from one digital image (disentangled from its appearance) and appearance from another from another digital image (disentangled from its content). For example, the disclosed systems can utilize a dual data augmentation method to generate diverse triplets for parameter learning (including input digital images, reference digital images, and pseudo ground truth digital images), via cropping a digital image with perturbations using three-dimensional color lookup tables ('LUTs'). Additionally, the disclosed systems can utilize the self-supervised image harmonization neural network to generate harmonized digital images that depict content from one digital image having the appearance of another digital image.

Learning parameters for an image harmonization neural network to generate deep harmonized digital images

The present disclosure relates to systems, methods, and non-transitory computer readable media that generate three-dimensional hybrid mesh-volumetric representations for digital objects. For instance, in one or more embodiments, the disclosed systems generate a mesh for a digital object from a plurality of digital images that portray the digital object using a multi-view stereo model. Additionally, the disclosed systems determine a set of sample points for a thin volume around the mesh. Using a neural network, the disclosed systems further generate a three-dimensional hybrid mesh-volumetric representation for the digital object utilizing the set of sample points for the thin volume and the mesh.

Generating three-dimensional representations for digital objects utilizing mesh-based thin volumes

Embodiments of the present disclosure provide systems, methods, and computer storage media for generating editable synthesized views of scenes by inputting image rays into neural networks using neural basis decomposition. In embodiments, a set of input images of a scene depicting at least one object are collected and used to generate a plurality of rays of the scene. The rays each correspond to three-dimensional coordinates and viewing angles taken from the images. A volume density of the scene is determined by inputting the three-dimensional coordinates from the neural radiance fields into a first neural network to generate a 3D geometric representation of the object. An appearance decomposition is produced by inputting the three-dimensional coordinates and the viewing angles of the rays into a second neural network.

Editing neural radiance fields with neural basis decomposition

A scene reconstruction system renders images of a scene with high-quality geometry and appearance and supports view synthesis, relighting, and scene editing. Given a set of input images of a scene, the scene reconstruction system trains a network to learn a volume representation of the scene that includes separate geometry and reflectance parameters. Using the volume representation, the scene reconstruction system can render images of the scene under arbitrary viewing (view synthesis) and lighting (relighting) locations. Additionally, the scene reconstruction system can render images that change the reflectance of objects in the scene (scene editing).

Scene reconstruction using geometry and reflectance volume representation of scene

The present disclosure relates to using end-to-end differentiable pipeline for optimizing parameters of a base procedural material to generate a procedural material corresponding to a target physical material. For example, the disclosed systems can receive a digital image of a target physical material. In response, the disclosed systems can retrieve a differentiable procedural material for use as a base procedural material in response. The disclosed systems can compare a digital image of the base procedural material with the digital image of the target physical material using a loss function, such as a style loss function that compares visual appearance. Based on the determined loss, the disclosed systems can modify the parameters of the base procedural material to determine procedural material parameters for the target physical material. The disclosed systems can generate a procedural material corresponding to the base procedural material using the determined procedural material parameters.

Generating differentiable procedural materials

The present disclosure relates to systems, non-transitory computer-readable media, and methods that implement a dual-branched neural network architecture to harmonize composite images. For example, in one or more implementations, the transformer-based harmonization system uses a convolutional branch and a transformer branch to generate a harmonized composite image based on an input composite image and a corresponding segmentation mask. More particularly, the convolutional branch comprises a series of convolutional neural network layers followed by a style normalization layer to extract localized information from the input composite image. Further, the transformer branch comprises a series of transformer neural network layers to extract global information based on different resolutions of the input composite image. Utilizing a decoder, the transformer-based harmonization system combines the local information and the global information from the corresponding convolutional branch and transformer branch to generate a harmonized composite image.

Harmonizing composite images utilizing a transformer neural network

The present disclosure relates to systems, methods, and non-transitory computer readable media for accurately and flexibly generating harmonized digital images utilizing an object-to-object harmonization neural network. For example, the disclosed systems implement, and learn parameters for, an object-to-object harmonization neural network to combine a style code from a reference object with features extracted from a target object. Indeed, the disclosed systems extract a style code from a reference object utilizing a style encoder neural network. In addition, the disclosed systems generate a harmonized target object by applying the style code of the reference object to a target object utilizing an object-to-object harmonization neural network.

Object-to-object harmonization for digital images

A scene modeling system receives a plurality of input two-dimensional (2D) images corresponding to a plurality of views of an object and a request to display a three-dimensional (3D) scene that includes the object. The scene modeling system generates an output 2D image for a view of the 3D scene by applying a scene representation model to the input 2D images. The scene representation model includes a point cloud generation model configured to generate, based on the input 2D images, a neural point cloud representing the 3D scene. The scene representation model includes a neural point volume rendering model configured to determine, for each pixel of the output image and using the neural point cloud and a volume rendering process, a color value. The scene modeling system transmits, responsive to the request, the output 2D image. Each pixel of the output image includes the respective determined color value.

Point-based neural radiance field for three dimensional scene representation

This disclosure relates to methods, non-transitory computer readable media, and systems that can render a virtual object in a digital image by using a source-specific-lighting-estimation-neural network to generate three-dimensional ('3D') lighting parameters specific to a light source illuminating the digital image. To generate such source-specific-lighting parameters, for instance, the disclosed systems utilize a compact source-specific-lighting-estimation-neural network comprising both common network layers and network layers specific to different lighting parameters. In some embodiments, the disclosed systems further train such a source-specific-lighting-estimation-neural network to accurately estimate spatially varying lighting in a digital image based on comparisons of predicted environment maps from a differentiable-projection layer with ground-truth-environment maps.

Generating light-source-specific parameters for digital images using a neural network

Methods, systems, and non-transitory computer readable storage media are disclosed for utilizing a visual neural network to replace materials in a three-dimensional scene with visually similar materials from a source dataset. Specifically, the disclosed system utilizes the visual neural network to generate source deep visual features representing source texture maps from materials in a plurality of source materials. Additionally, the disclosed system utilizes the visual neural network to generate deep visual features representing texture maps from materials in a digital scene. The disclosed system then determines source texture maps that are visually similar to the texture maps of the digital scene based on visual similarity metrics that compare the source deep visual features and the deep visual features. Additionally, the disclosed system modifies the digital scene by replacing one or more of the texture maps in the digital scene with the visually similar source texture maps.

Modifying materials of three-dimensional digital scenes utilizing a visual neural network

Generating deep harmonized digital images

Systems and techniques for automatic digital parameter adjustment are described that leverage insights learned from an image set to automatically predict parameter values for an input item of digital visual content. To do so, the automatic digital parameter adjustment techniques described herein captures visual and contextual features of digital visual content to determine balanced visual output in a range of visual scenes and settings. The visual and contextual features of digital visual content are used to train a parameter adjustment model through machine learning techniques that captures feature patterns and interactions. The parameter adjustment model exploits these feature interactions to determine visually pleasing parameter values for an input item of digital visual content. The predicted parameter values are output, allowing further adjustment to the parameter values.

Automated digital parameter adjustment for digital images

Embodiments are disclosed for neural texture mapping. In some embodiments, a method of neural texture mapping includes obtaining a plurality of images of an object, determining volumetric representation of a scene of the object using a first neural network, mapping 3D points of the scene to a 2D texture space using a second neural network, and determining radiance values for each 2D point in the 2D texture space from a plurality of viewpoints using a second neural network to generate a 3D appearance representation of the object.

Learning 2D texture mapping in volumetric neural rendering

Growing community of inventors

Cambridge, MA, United States of America

Kalyan Krishna Sunkavalli