Adobe, Inc.

He Zhang

Jianming Zhang

Zhe Lin

Yilin Wang

Zijun Wei

Brian Lynn Price

Kalyan Krishna Sunkavalli

Elya Shechtman

Seyed Morteza Safdarnejad

Salil Tambe

Chenglin Yang

Sohrab Amirghodsi

Ning Xu

Jose Ignacio Echevarria Vallespi

Sarah Kong

Federico Perazzi

Su Chen

Yinglan Ma

Yifan Jiang

Yutong Dai

Qihang Yu

Xin Sun

Julien Philip

Yannick Hold-Geoffroy

Meredith Payne Stotzner

Frederick Mandia

Hyun Joon Jung

Ke Wang

Jeya Maria Jose Valanarasu

Yifan Liu

Yichen Sheng

An image processing system employs a parametric model for image harmonization of composite images. The parametric model employs a two-stage approach to harmonize an input composite image. At a first stage, a color curve prediction model predicts color curve parameters for the composite image. At a second stage, the composite image with the color curve parameters is input to a shadow map prediction model, which predicts a shadow map. The predicted color curve parameters and shadow map are applied to the composite image to provide a harmonized composite image. In some aspects, the color curve parameters and shadow map are predicted using a lower-resolution composite image and up-sampled to apply to a higher-resolution version of the composite image. The harmonized composite image can be output with the predicted color curve parameters and/or shadow map, which can be modified by a user to further enhance the harmonized composite image.

Parametric composite image harmonization

The present disclosure relates to systems, methods, and non-transitory computer-readable media that generates object shadows for digital images utilizing corresponding geometry-aware buffer channels. For instance, in one or more embodiments, the disclosed systems generate, utilizing a height prediction neural network, an object height map for a digital object portrayed in a digital image and a background height map for a background portrayed in the digital image. The disclosed systems also generate, from the digital image, a plurality of geometry-aware buffer channels using the object height map and the background height map. Further, the disclosed systems modify the digital image to include a soft object shadow for the digital object using the plurality of geometry-aware buffer channels.

Generating soft object shadows for general shadow receivers within digital images using geometry-aware buffer channels

Systems and methods for image segmentation are described. Embodiments of the present disclosure receive an image depicting an object; generate image features for the image by performing a convolutional self-attention operation that outputs a plurality of attention-weighted values for a convolutional kernel applied at a position of a sliding window on the image; and generate label data that identifies the object based on the image features.

Transformer for efficient image segmentation

The present disclosure relates to systems, non-transitory computer-readable media, and methods that utilize a progressive refinement network to refine alpha mattes generated utilizing a mask-guided matting neural network. In particular, the disclosed systems can use the matting neural network to process a digital image and a coarse guidance mask to generate alpha mattes at discrete neural network layers. In turn, the disclosed systems can use the progressive refinement network to combine alpha mattes and refine areas of uncertainty. For example, the progressive refinement network can combine a core alpha matte corresponding to more certain core regions of a first alpha matte and a boundary alpha matte corresponding to uncertain boundary regions of a second, higher resolution alpha matte. Based on the combination of the core alpha matte and the boundary alpha matte, the disclosed systems can generate a final alpha matte for use in image matting processes.

Generating alpha mattes utilizing deep learning

Systems and methods for image segmentation are described. Embodiments of the present disclosure receive an image depicting an object; generate image features for the image by performing an atrous self-attention operation based on a plurality of dilation rates for a convolutional kernel applied at a position of a sliding window on the image; and generate label data that identifies the object based on the image features.

The present disclosure relates to systems, methods, and non-transitory computer readable media for accurately, efficiently, and flexibly generating harmonized digital images utilizing a self-supervised image harmonization neural network. In particular, the disclosed systems can implement, and learn parameters for, a self-supervised image harmonization neural network to extract content from one digital image (disentangled from its appearance) and appearance from another from another digital image (disentangled from its content). For example, the disclosed systems can utilize a dual data augmentation method to generate diverse triplets for parameter learning (including input digital images, reference digital images, and pseudo ground truth digital images), via cropping a digital image with perturbations using three-dimensional color lookup tables ('LUTs'). Additionally, the disclosed systems can utilize the self-supervised image harmonization neural network to generate harmonized digital images that depict content from one digital image having the appearance of another digital image.

Learning parameters for an image harmonization neural network to generate deep harmonized digital images

An image processing system generates an image mask from an image. The image is processed by an object detector to identify a region having an object, and the region is classified based on an object type of the object. A masking pipeline is selected from a number of masking pipelines based on the classification of the region. The region is processed using the masking pipeline to generate a region mask. An image mask for the image is generated using the region mask.

Object selection for images using image regions

This disclosure describes one or more implementations of an alpha matting system that utilizes a deep learning model to generate alpha mattes for digital images utilizing an alpha-range classifier function. More specifically, in various implementations, the alpha matting system builds and utilizes an object mask neural network having a decoder that includes an alpha-range classifier to determine classification probabilities for pixels of a digital image with respect to multiple alpha-range classifications. In addition, the alpha matting system can utilize a refinement model to generate the alpha matte from the pixel classification probabilities with respect to the multiple alpha-range classifications.

Generating improved alpha mattes for digital images based on pixel classification probabilities across alpha-range classifications

The present disclosure relates to systems, non-transitory computer-readable media, and methods that implement a multi-branch harmonization neural network architecture to harmonize composite images. For example, in one or more implementations, the semantic-guided transformer-based harmonization system uses a convolutional branch, a transformer branch, and a semantic branch to generate a harmonized composite image based on an input composite image and a corresponding segmentation mask. More particularly, the convolutional branch comprises a series of convolutional neural network layers followed by a style normalization layer to extract localized information from the input composite image. Further, the transformer branch comprises a series of transformer neural network layers to extract global information based on different resolutions of the input composite image. The semantic branch includes a visual neural network that generates semantic features that inform the harmonization of the composite images.

Harmonizing composite images utilizing a semantic-guided transformer neural network

The present disclosure relates to systems, methods, and non-transitory computer-readable media that generate a height map for a digital object portrayed in a digital image and further utilizes the height map to generate a shadow for the digital object. Indeed, in one or more embodiments, the disclosed systems generate (e.g., utilizing a neural network) a height map that indicates the pixels heights for pixels of a digital object portrayed in a digital image. The disclosed systems utilize the pixel heights, along with lighting information for the digital image, to determine how the pixels of the digital image project to create a shadow for the digital object. Further, in some implementations, the disclosed systems utilize the determined shadow projections to generate (e.g., utilizing another neural network) a soft shadow for the digital object. Accordingly, in some cases, the disclosed systems modify the digital image to include the shadow.

Generating shadows for digital objects within digital images utilizing a height map

Methods, systems, and non-transitory computer readable storage media are disclosed for utilizing a plurality of neural networks in a multi-branch pipeline to generate image masks for digital images. Specifically, the disclosed system can classify a digital image as a portrait or a non-portrait image. Based on classifying a portrait image, the disclosed system can utilize separate neural networks to generate a first mask portion for a portion of the digital image including a defined boundary region and a second mask portion for a portion of the digital image including a blended boundary region. The disclosed system can generate the mask portion for the blended boundary region by utilizing a trimap generation neural network to automatically generate a trimap segmentation including the blended boundary region. The disclosed system can then merge the first mask portion and the second mask portion to generate an image mask for the digital image.

Generating an image mask for a digital image by utilizing a multi-branch masking pipeline with neural networks

The present disclosure relates to systems, non-transitory computer-readable media, and methods that implement a dual-branched neural network architecture to harmonize composite images. For example, in one or more implementations, the transformer-based harmonization system uses a convolutional branch and a transformer branch to generate a harmonized composite image based on an input composite image and a corresponding segmentation mask. More particularly, the convolutional branch comprises a series of convolutional neural network layers followed by a style normalization layer to extract localized information from the input composite image. Further, the transformer branch comprises a series of transformer neural network layers to extract global information based on different resolutions of the input composite image. Utilizing a decoder, the transformer-based harmonization system combines the local information and the global information from the corresponding convolutional branch and transformer branch to generate a harmonized composite image.

Harmonizing composite images utilizing a transformer neural network

The present disclosure relates to systems, methods, and non-transitory computer readable media for accurately and flexibly generating harmonized digital images utilizing an object-to-object harmonization neural network. For example, the disclosed systems implement, and learn parameters for, an object-to-object harmonization neural network to combine a style code from a reference object with features extracted from a target object. Indeed, the disclosed systems extract a style code from a reference object utilizing a style encoder neural network. In addition, the disclosed systems generate a harmonized target object by applying the style code of the reference object to a target object utilizing an object-to-object harmonization neural network.

Object-to-object harmonization for digital images

Methods, systems, and non-transitory computer readable storage media are disclosed for utilizing a transformer-based encoder-decoder neural network architecture for generating alpha mattes for digital images. Specifically, the disclosed system utilizes a transformer encoder to generate patch-based encodings from a digital image and a trimap segmentation by generating patch encodings for image patches and comparing the patch encodings to areas of the digital image. Additionally, the disclosed system generates modified patch-based encodings utilizing a plurality of neural network layers. The disclosed system also generates an alpha matte for the digital image from the patch-based encodings utilizing a decoder that includes a plurality of upsampling layers connected to a plurality of neural network layers via skip connections. In additional embodiments, the disclosed system generates the alpha matte based on additional encodings generated by a plurality of convolutional neural network layers connected to a subset of the upsampling layers via skip connections.

Growing community of inventors

San Jose, CA, United States of America

He Zhang