Adobe, Inc.

Adobe Systems Inc.

Other

Czech Technical University in Prague

Elya Shechtman

Jingwan Lu

Sohrab Amirghodsi

Connelly Barnes

Oliver Wang

Richard Zhang

Kalyan Krishna Sunkavalli

Zhe Lin

Aliakbar Darabi

Ratheesh Kalarot

Jue Wang

Jianming Zhang

Daniel Goldman

Shabnam Ghadar

Dan Goldman

Sylvain Philippe Paris

Michal Lukac

Geoffrey Oxholm

Wei-An Lin

Nathan Aaron Carr

Yijun Li

Zhixin Shu

Taesung Park

Paul John Asente

Jakub Fiser

Alexei A Efros

Jun-Yan Zhu

Ondrej Jamriska

Matthew David Fisher

Ning Xu

Cameron Smith

Haitian Zheng

Daniel Sýkora

Hyunjoon Lee

Scott David Cohen

Baldo Antonio Faieta

Aseem Agarwala

Niloy Jyoti Mitra

Seungyong Lee

Junyan Zhu

Yuqian Zhou

Chetan Nanda

Hailin Jin

Jianchao Yang

Jonathan Worthen Brandt

Jose Ignacio Echevarria Vallespi

Aaron Phillip Hertzmann

Vladimir Kim

Mehmet Yumer

Kevin Wampler

Chen-Hsuan Lin

Sunil Sharadchandra Hadap

Jimei Yang

Lubomir D Bourdev

David P Simons

Yang Zhou

Jonathan Eisenmann

Bryan Russell

Krishna Kumar Singh

Adam Finkelstein

John Nelson

Qing Liu

Ian D Sachs

John Thomas Nack

Michal Lukác

Guang Chen

Tharun Mohandoss

Wenqi Xian

Yu Zeng

Pulkit Gera

Fei Yang

Yuheng Li

Kartik Sethi

Brian Lynn Price

Zhaowen Wang

Yilin Wang

Xue Bai

Dingzeyu Li

He Zhang

Alexandru Vasile Costin

Hui Qu

Sarah Stuckey

Shai Bagon

David Edward Jacobs

Xiaoyong Shen

Chen Fang

Duygu Ceylan Aksit

Saeid Motiian

Jun Saito

Zijun Wei

Daniel Lischinski

Akhilesh Kumar

Yoav HaCohen

Michael Gharbi

Yinglan Ma

Florian Kainz

Omid Poursaeed

Siddhant Jain

Zoya Bylinskii

Systems and methods train an encoder neural network for fast and accurate projection into the latent space of a Generative Adversarial Network (GAN). The encoder is trained by providing an input training image to the encoder and producing, by the encoder, a latent space representation of the input training image. The latent space representation is provided as input to the GAN to generate a generated training image. A latent code is sampled from a latent space associated with the GAN and the sampled latent code is provided as input to the GAN. The GAN generates a synthetic training image based on the sampled latent code. The sampled latent code is provided as input to the encoder to produce a synthetic training code. The encoder is updated by minimizing a loss between the generated training image and the input training image, and the synthetic training code and the sampled latent code.

Supervised learning techniques for encoder training

The present disclosure relates to systems, methods, and non-transitory computer readable media that generate inpainted digital images utilizing class-specific cascaded modulation inpainting neural network. For example, the disclosed systems utilize a class-specific cascaded modulation inpainting neural network that includes cascaded modulation decoder layers to generate replacement pixels portraying a particular target object class. To illustrate, in response to user selection of a replacement region and target object class, the disclosed systems utilize a class-specific cascaded modulation inpainting neural network corresponding to the target object class to generate an inpainted digital image that portrays an instance of the target object class within the replacement region. Moreover, in one or more embodiments the disclosed systems train class-specific cascaded modulation inpainting neural networks corresponding to a variety of target object classes, such as a sky object class, a water object class, a ground object class, or a human object class.

Object class inpainting in digital images utilizing class-specific inpainting neural networks

The present disclosure relates to systems, methods, and non-transitory computer readable media for generating modified digital images by utilizing a patch match algorithm to generate nearest neighbor fields for a second digital image based on a nearest neighbor field associated with a first digital image. For example, the disclosed systems can identify a nearest neighbor field associated with a first digital image of a first resolution. Based on the nearest neighbor field of the first digital image, the disclosed systems can utilize a patch match algorithm to generate a nearest neighbor field for a second digital image of a second resolution larger than the first resolution. The disclosed systems can further generate a modified digital image by filling a target region of the second digital image utilizing the generated nearest neighbor field.

Generating modified digital images utilizing nearest neighbor fields from patch matching operations of alternate digital images

The present disclosure relates to systems, methods, and non-transitory computer readable media for panoptically guiding digital image inpainting utilizing a panoptic inpainting neural network. In some embodiments, the disclosed systems utilize a panoptic inpainting neural network to generate an inpainted digital image according to panoptic segmentation map that defines pixel regions corresponding to different panoptic labels. In some cases, the disclosed systems train a neural network utilizing a semantic discriminator that facilitates generation of digital images that are realistic while also conforming to a semantic segmentation. The disclosed systems generate and provide a panoptic inpainting interface to facilitate user interaction for inpainting digital images. In certain embodiments, the disclosed systems iteratively update an inpainted digital image based on changes to a panoptic segmentation map.

Generating and providing a panoptic inpainting interface for generating and modifying inpainted digital images

Iteratively modifying inpainted digital images based on changes to panoptic segmentation maps

Learning parameters for neural networks using a semantic discriminator and an object-level discriminator

Systems and methods for image generation are provided. An aspect of the systems and methods for image generation includes obtaining an original image depicting an element and a target prompt describing a modification to the element. The system may then compute a first output and a second output using a diffusion model. The first output is based on a description of the element and the second output is based on the target prompt. The system then computes a difference between the first output and the second output, and generates a modified image including the modification to the element of the original image based on the difference.

Image generation using a diffusion model

An improved system architecture uses a Generative Adversarial Network (GAN) including a specialized generator neural network to generate multiple resolution output images. The system produces a latent space representation of an input image. The system generates a first output image at a first resolution by providing the latent space representation of the input image as input to a generator neural network comprising an input layer, an output layer, and a plurality of intermediate layers and taking the first output image from an intermediate layer, of the plurality of intermediate layers of the generator neural network. The system generates a second output image at a second resolution different from the first resolution by providing the latent space representation of the input image as input to the generator neural network and taking the second output image from the output layer of the generator neural network.

Multi-scale output techniques for generative adversarial networks

Systems and methods train and apply a specialized encoder neural network for fast and accurate projection into the latent space of a Generative Adversarial Network (GAN). The specialized encoder neural network includes an input layer, a feature extraction layer, and a bottleneck layer positioned after the feature extraction layer. The projection process includes providing an input image to the encoder and producing, by the encoder, a latent space representation of the input image. Producing the latent space representation includes extracting a feature vector from the feature extraction layer, providing the feature vector lo the bottleneck layer as input, and producing the latent space representation as output. The latent space representation produced by the encoder is provided as input to the GAN, which generates an output image based upon the latent space representation.

Direct regression encoder architecture and training

One or more processing devices access a scene depicting a reference object that includes an annotation identifying a target region to be modified in one or more video frames. The one or more processing devices determine that a target pixel corresponds to a sub-region within the target region that includes hallucinated content. The one or more processing devices determine gradient constraints using gradient values of neighboring pixels in the hallucinated content, the neighboring pixels being adjacent to the target pixel and corresponding to four cardinal directions. The one or more processing devices update color data of the target pixel subject to the determined gradient constraints.

Corrective lighting for video inpainting

Systems and methods for image generation include obtaining an input image and an attribute value representing an attribute of the input image to be modified; computing a modified latent vector for the input image by applying the attribute value to a basis vector corresponding to the attribute in a latent space of an image generation network; and generating a modified image based on the modified latent vector using the image generation network, wherein the modified image includes the attribute based on the attribute value.

Product of variations in image generative models

An image inpainting system is described that receives an input image that includes a masked region. From the input image, the image inpainting system generates a synthesized image that depicts an object in the masked region by selecting a first code that represents a known factor characterizing a visual appearance of the object and a second code that represents an unknown factor characterizing the visual appearance of the object apart from the known factor in latent space. The input image, the first code, and the second code are provided as input to a generative adversarial network that is trained to generate the synthesized image using contrastive losses. Different synthesized images are generated from the same input image using different combinations of first and second codes, and the synthesized images are output for display.

Diverse image inpainting using contrastive learning

Embodiments are disclosed for performing wire segmentation of images using machine learning. In particular, in one or more embodiments, the disclosed systems and methods comprise receiving an input image, generating, by a first trained neural network model, a global probability map representation of the input image indicating a probability value of each pixel including a representation of wires, and identifying regions of the input image indicated as including the representation of wires. The disclosed systems and methods further comprise, for each region from the identified regions, concatenating the region and information from the global probability map to create a concatenated input, and generating, by a second trained neural network model, a local probability map representation of the region based on the concatenated input, indicating pixels of the region including representations of wires. The disclosed systems and methods further comprise aggregating local probability maps for each region.

Wire segmentation for images using machine learning

Methods, systems, and non-transitory computer readable media are disclosed for intelligently enhancing details in edited images. The disclosed system iteratively updates residual detail latent code for segments in edited images where detail has been lost through the editing process. More particularly, the disclosed system enhances an edited segment in an edited image based on details in a detailed segment of an image. Additionally, the disclosed system may utilize a detail neural network encoder to project the detailed segment and a corresponding segment of the edited image into a residual detail latent code. In some embodiments, the disclosed system generates a refined edited image based on the residual detail latent code and a latent vector of the edited image.

Enhancing detailed segments in latent code-based edited digital images

The present disclosure relates to systems, methods, and non-transitory computer readable media for accurately and flexibly generating modified digital images utilizing a novel swapping autoencoder that incorporates scene layout. In particular, the disclosed systems can receive a scene layout map that indicates or defines locations for displaying specific digital content within a digital image. In addition, the disclosed systems can utilize the scene layout map to guide combining portions of digital image latent code to generate a modified digital image with a particular textural appearance and a particular geometric structure defined by the scene layout map. Additionally, the disclosed systems can utilize a scene layout map that defines a portion of a digital image to modify by, for instance, adding new digital content to the digital image, and can generate a modified digital image depicting the new digital content.

Generating modified digital images incorporating scene layout utilizing a swapping autoencoder

Growing community of inventors

Seattle, WA, United States of America

Elya Shechtman