Adobe, Inc.

Richard Zhang

Elya Shechtman

Jingwan Lu

Taesung Park

Shabnam Ghadar

Ratheesh Kalarot

Junyan Zhu

Jun-Yan Zhu

Yijun Li

Zhixin Shu

Alexei A Efros

Niloy Jyoti Mitra

Cameron Smith

Wei-An Lin

Oliver Wang

Baldo Antonio Faieta

Matthew David Fisher

Sylvain Philippe Paris

Zhe Lin

Nathan Aaron Carr

Aaron Phillip Hertzmann

Aliakbar Darabi

Krishna Kumar Singh

Tobias Hinz

Jacob Minyoung Huh

Michaël Gharbi

Yannick Hold-Geoffrey

Gaurav Parmar

Yang Zhou

Saeid Motiian

Bryan Russell

Zhifei Zhang

Sameer Bhatt

Ionut Mironica

Nishant Kumar

Siavash Khodadadeh

Vikas Sharma

Evangelos Kalogerakis

Yotam Nitzan

Difan Liu

Yuchen Liu

Rupali Arora

Anuradha Yadav

William Peebles

Alyosha Efros

Utkarsh Ojha

Seungjoo Yoo

Nadav Epstein

Adrian-Stefan Ungureanu

Yinbo Chen

Xiaojuan Wang

Alexander Andonian

Shantanu Agarwal

Aspects of the methods, apparatus, non-transitory computer readable medium, and systems include obtaining a noise map and a global image code encoded from an original image and representing semantic content of the original image; generating a plurality of image patches based on the noise map and the global image code using a diffusion model; and combining the plurality of image patches to produce an output image including the semantic content.

Diffusion models having continuous scaling through patch-wise image generation

An image processing system uses a depth-conditioned autoencoder to generate a modified image from an input image such that the modified image maintains an overall structure from the input image while modifying textural features. An encoder of the depth-conditioned autoencoder extracts a structure latent code from an input image and depth information for the input image. A generator of the depth-conditioned autoencoder generates a modified image using the structure latent code and a texture latent code. The modified image generated by the depth-conditioned autoencoder includes the structural features from the input image while incorporating textural features of the texture latent code. In some aspects, the autoencoder is depth-conditioned during training by augmenting training images with depth information. The autoencoder is trained to preserve the depth information when generating images.

Generation using depth-conditioned autoencoder

Semantic fill techniques are described that support generating fill and editing images from semantic inputs. A user input, for example, is received by a semantic fill system that indicates a selection of a first region of a digital image and a corresponding semantic label. The user input is utilized by the semantic fill system to generate a guidance attention map of the digital image. The semantic fill system leverages the guidance attention map to generate a sparse attention map of a second region of the digital image. A semantic fill of pixels is generated for the first region based on the semantic label and the sparse attention map. The edited digital image is displayed in a user interface.

Semantic image fill at high resolutions

Systems and methods train an encoder neural network for fast and accurate projection into the latent space of a Generative Adversarial Network (GAN). The encoder is trained by providing an input training image to the encoder and producing, by the encoder, a latent space representation of the input training image. The latent space representation is provided as input to the GAN to generate a generated training image. A latent code is sampled from a latent space associated with the GAN and the sampled latent code is provided as input to the GAN. The GAN generates a synthetic training image based on the sampled latent code. The sampled latent code is provided as input to the encoder to produce a synthetic training code. The encoder is updated by minimizing a loss between the generated training image and the input training image, and the synthetic training code and the sampled latent code.

Supervised learning techniques for encoder training

Techniques for identity preserved controllable facial image manipulation are described that support generation of a manipulated digital image based on a facial image and a render image. For instance, a facial image depicting a facial representation of an individual is received as input. A feature space including an identity parameter and at least one other visual parameter is extracted from the facial image. An editing module edits one or more of the visual parameters and preserves the identity parameter. A renderer generates a render image depicting a morphable model reconstruction of the facial image based on the edit. The render image and facial image are encoded, and a generator of a neural network is implemented to generate a manipulated digital image based on the encoded facial image and the encoded render image.

Identity preserved controllable facial image manipulation

An improved system architecture uses a Generative Adversarial Network (GAN) including a specialized generator neural network to generate multiple resolution output images. The system produces a latent space representation of an input image. The system generates a first output image at a first resolution by providing the latent space representation of the input image as input to a generator neural network comprising an input layer, an output layer, and a plurality of intermediate layers and taking the first output image from an intermediate layer, of the plurality of intermediate layers of the generator neural network. The system generates a second output image at a second resolution different from the first resolution by providing the latent space representation of the input image as input to the generator neural network and taking the second output image from the output layer of the generator neural network.

Multi-scale output techniques for generative adversarial networks

The present disclosure relates to systems, non-transitory computer-readable media, and methods for utilizing machine learning models to generate modified digital images. In particular, in some embodiments, the disclosed systems generate image editing directions between textual identifiers of two visual features utilizing a language prediction machine learning model and a text encoder. In some embodiments, the disclosed systems generated an inversion of a digital image utilizing a regularized inversion model to guide forward diffusion of the digital image. In some embodiments, the disclosed systems utilize cross-attention guidance to preserve structural details of a source digital image when generating a modified digital image with a diffusion neural network.

Utilizing cross-attention guidance to preserve content in diffusion-based image modifications

Systems and methods train and apply a specialized encoder neural network for fast and accurate projection into the latent space of a Generative Adversarial Network (GAN). The specialized encoder neural network includes an input layer, a feature extraction layer, and a bottleneck layer positioned after the feature extraction layer. The projection process includes providing an input image to the encoder and producing, by the encoder, a latent space representation of the input image. Producing the latent space representation includes extracting a feature vector from the feature extraction layer, providing the feature vector lo the bottleneck layer as input, and producing the latent space representation as output. The latent space representation produced by the encoder is provided as input to the GAN, which generates an output image based upon the latent space representation.

Direct regression encoder architecture and training

Systems and methods for image generation include obtaining an input image and an attribute value representing an attribute of the input image to be modified; computing a modified latent vector for the input image by applying the attribute value to a basis vector corresponding to the attribute in a latent space of an image generation network; and generating a modified image based on the modified latent vector using the image generation network, wherein the modified image includes the attribute based on the attribute value.

Product of variations in image generative models

The present disclosure relates to systems, methods, and non-transitory computer readable media for accurately and flexibly generating modified digital images utilizing a novel swapping autoencoder that incorporates scene layout. In particular, the disclosed systems can receive a scene layout map that indicates or defines locations for displaying specific digital content within a digital image. In addition, the disclosed systems can utilize the scene layout map to guide combining portions of digital image latent code to generate a modified digital image with a particular textural appearance and a particular geometric structure defined by the scene layout map. Additionally, the disclosed systems can utilize a scene layout map that defines a portion of a digital image to modify by, for instance, adding new digital content to the digital image, and can generate a modified digital image depicting the new digital content.

Generating modified digital images incorporating scene layout utilizing a swapping autoencoder

An image generation system enables user input during the process of training a generative model to influence the model's ability to generate new images with desired visual features. A source generative model for a source domain is fine-tuned using training images in a target domain to provide an adapted generative model for the target domain. Interpretable factors are determined for the source generative model and the adapted generative model. A user interface is provided that enables a user to select one or more interpretable factors. The user-selected interpretable factor(s) are used to generate a user-adapted generative model, for instance, by using a loss function based on the user-selected interpretable factor(s). The user-adapted generative model can be used to create new images in the target domain.

User-guided image generation

The present disclosure relates to systems, non-transitory computer-readable media, and methods for combining digital images. In particular, in one or more embodiments, the disclosed systems combine latent codes of a source digital image and a target digital image utilizing a blending network to determine a combined latent encoding and generate a combined digital image from the combined latent encoding utilizing a generative neural network. In some embodiments, the disclosed systems determine an intersection face mask between the source digital image and the combined digital image utilizing a face segmentation network and combine the source digital image and the combined digital image utilizing the intersection face mask to generate a blended digital image.

Transferring faces between digital images by combining latent codes utilizing a blending network

The present disclosure relates to systems, methods, and non-transitory computer-readable media that utilize machine learning to generate a sequence of transition frames for a gap in a clipped digital video. For example, the disclosed system receives a clipped digital video that includes a pre-cut frame prior to a gap in the clipped digital video and a post-cut frame following the gap in the clipped digital video. Moreover, the disclosed system utilizes a natural motion sequence model to generates a sequence of transition keypoint maps between the pre-cut frame and the post-cut frame. Furthermore, using a generative neural network, the disclosed system generates a sequence of transition frames for the gap in the clipped digital video from the sequence of transition keypoint maps.

Utilizing generative models for resynthesis of transition frames in clipped digital videos

In implementations of systems for image inversion using multiple latent spaces, a computing device implements an inversion system to generate a segment map that segments an input digital image into a first image region and a second image region and assigns the first image region to a first latent space and the second image region to a second latent space that corresponds to a layer of a convolutional neural network. An inverted latent representation of the input digital image is computed using a binary mask for the second image region. The inversion system modifies the inverted latent representation of the input digital image using an edit direction vector that corresponds to a visual feature. An output digital image is generated that depicts a reconstruction of the input digital image having the visual feature based on the modified inverted latent representation of the input digital image.

Image inversion using multiple latent spaces

The present disclosure relates to systems, methods, and non-transitory computer readable media for generating digital images depicting photorealistic scenes utilizing a digital image collaging neural network. For example, the disclosed systems utilize a digital image collaging neural network having a particular architecture for disentangling generation of scene layouts and pixel colors for different regions of a digital image. In some cases, the disclosed systems break down the process of generating a collage digital into generating images representing different regions such as a background and a foreground to be collaged into a final result. For example, utilizing the digital image collaging neural network, the disclosed systems determine scene layouts and pixel colors for both foreground digital images and background digital images to ultimately collage the foreground and background together into a collage digital image depicting a real-world scene.

Generating collage digital images by combining scene layouts and pixel colors utilizing generative neural networks

Growing community of inventors

San Francisco, CA, United States of America

Richard Zhang