Adobe, Inc.

Microsoft Technology Licensing, LLC

Junyan Zhu

Richard Zhang

Taesung Park

Elya Shechtman

Alexei A Efros

Jingwan Lu

Oliver Wang

Sylvain Philippe Paris

Aaron Phillip Hertzmann

Eric I-Chao Chang

Zhuowen Tu

Jacob Minyoung Huh

Yan Xu

Bryan Russell

Yichen Wei

Michaël Gharbi

Yotam Nitzan

Jiajun Wu

Alexander Andonian

Systems and methods for image generation include obtaining an input image and an attribute value representing an attribute of the input image to be modified; computing a modified latent vector for the input image by applying the attribute value to a basis vector corresponding to the attribute in a latent space of an image generation network; and generating a modified image based on the modified latent vector using the image generation network, wherein the modified image includes the attribute based on the attribute value.

Product of variations in image generative models

The present disclosure relates to systems, methods, and non-transitory computer readable media for accurately and flexibly generating modified digital images utilizing a novel swapping autoencoder that incorporates scene layout. In particular, the disclosed systems can receive a scene layout map that indicates or defines locations for displaying specific digital content within a digital image. In addition, the disclosed systems can utilize the scene layout map to guide combining portions of digital image latent code to generate a modified digital image with a particular textural appearance and a particular geometric structure defined by the scene layout map. Additionally, the disclosed systems can utilize a scene layout map that defines a portion of a digital image to modify by, for instance, adding new digital content to the digital image, and can generate a modified digital image depicting the new digital content.

Generating modified digital images incorporating scene layout utilizing a swapping autoencoder

The present disclosure relates to systems, methods, and non-transitory computer readable media for generating a modified digital image from extracted spatial and global codes. For example, the disclosed systems can utilize a global and spatial autoencoder to extract spatial codes and global codes from digital images. The disclosed systems can further utilize the global and spatial autoencoder to generate a modified digital image by combining extracted spatial and global codes in various ways for various applications such as style swapping, style blending, and attribute editing.

Generating modified digital images utilizing a global and spatial autoencoder

A target image is projected into a latent space of generative model by determining a latent vector by applying a gradient-free technique and a class vector by applying a gradient-based technique. An image is generated from the latent and class vectors, and a loss function is used to determine a loss between the target image and the generated image. This determining of the latent vector and the class vector, generating an image, and using the loss function is repeated until a loss condition is satisfied. In response to the loss condition being satisfied, the latent and class vectors that resulted in the loss condition being satisfied are identified as the final latent and class vectors, respectively. The final latent and class vectors are provided to the generative model and multiple weights of the generative model are adjusted to fine-tune the generative model.

Projecting images to a generative model based on gradient-free latent vector determination

This disclosure describes methods, non-transitory computer readable storage media, and systems that utilize a contrastive perceptual loss to modify neural networks for generating synthetic digital content items. For example, the disclosed systems generate a synthetic digital content item based on a guide input to a generative neural network. The disclosed systems utilize an encoder neural network to generate encoded representations of the synthetic digital content item and a corresponding ground-truth digital content item. Additionally, the disclosed systems sample patches from the encoded representations of the encoded digital content items and then determine a contrastive loss based on the perceptual distances between the patches in the encoded representations. Furthermore, the disclosed systems jointly update the parameters of the generative neural network and the encoder neural network utilizing the contrastive loss.

Modifying neural networks for synthetic conditional digital content generation utilizing contrastive perceptual loss

The techniques and systems described herein create and train a multiple clustered instance learning (MCIL) model based on image features and patterns extracted from training images. The techniques and systems separate each of the training images into a plurality of instances (or patches), and then learn multiple instance-level classifiers based on the extracted image features. The instance-level classifiers are then integrated into the MCIL model so that the MCIL model, when applied to unclassified images, can perform image-level classification, patch-level clustering, and pixel-level segmentation.

Multiple cluster instance learning for image classification

Techniques for unsupervised object class discovery via bottom-up multiple class learning are described. These techniques may include receiving multiple images containing one or more object classes. The multiple images may be analyzed to extract top saliency instances and least saliency instances. These saliency instances may be clustered to generate and/or update statistical models. The statistical models may be used to discover the one or more object classes. In some instances, the statistical models may be used to discover object classes of novel images.

Unsupervised object class discovery via bottom up multiple class learning

Growing community of inventors

Cambridge, MA, United States of America

Junyan Zhu