Adobe, Inc.

Adobe Systems Inc.

Adobeinc.

Splunk Inc.

Jianming Zhang

Zhe Lin

Xiaohui Shen

Yilin Wang

He Zhang

Radomir Mech

Scott David Cohen

Elya Shechtman

Jingwan Lu

Zijun Wei

Ning Xu

Jimei Yang

Sohrab Amirghodsi

Haitian Zheng

Brian Lynn Price

Jonathan Worthen Brandt

Mingyang Ling

Qing Liu

Oliver Wang

Federico Perazzi

Jason Wen Yong Kuen

Su Chen

Kalyan Krishna Sunkavalli

Jen-Chan Jeff Chien

Baldo Antonio Faieta

Connelly Barnes

Krishna Kumar Singh

Lu Zhang

Jose Ignacio Echevarria Vallespi

Xin Sun

Chen Fang

Sarah Kong

Anant Gilra

Joon-Young Lee

Alex Charles Filipkowski

Seyed Morteza Safdarnejad

Simon Niklaus

Mai Long

Brett Michael Butterfield

Salil Tambe

Siyuan Qiao

Jason Kuen

Chenglin Yang

Shikun Liu

Yu Zeng

Hailin Jin

Xin Lu

Quan Anh Tran

Duygu Ceylan Aksit

Yijun Li

Handong Zhao

Zhihong Ding

I-Ming Pao

Ruben Eduardo Villegas

Yangtuanfeng Wang

Samarth Gulati

Yinglan Ma

Lijun Wang

Betty Leong

Tobias Hinz

Yingwei Li

Yifan Jiang

Wei Yin

Manuel Lagunas Arto

Xihui Liu

Qihang Yu

Steve Yu Zhang

Ajinkya Gorakhnath Kale

Alexandros Batsakis

Aaron Phillip Hertzmann

Jayant Kumar

Dingzeyu Li

Ratheesh Kalarot

Yannick Hold-Geoffroy

Alan Lee Erickson

Zhifei Zhang

John Philip Collomosse

Sanyam Jain

Haoxiang Li

Midhun Harikumar

Akhilesh Kumar

Yuqian Zhou

Hyunghwan Byun

Daniel Miranda

Balaji Rao

Pramod Srinivasan

Yufei Wang

Kerem Can Turgutlu

Amish Kumar Bedi

Nitilaksha Halakatti

Sarah Stuckey

Julien Philip

Chenxi Liu

Yukie Takahashi

Manuel Martinez

Ningxuan He

Klaas Stoeckmann

Thomas Muehrke

Jinrong Xie

Guotong Feng

Systems and methods for image processing are described. Embodiments of the present disclosure receive an image comprising a first region that includes content and a second region to be inpainted. Noise is then added to the image to obtain a noisy image, and a plurality of intermediate output images are generated based on the noisy image using a diffusion model trained using a perceptual loss. The intermediate output images predict a final output image based on a corresponding intermediate noise level of the diffusion model. The diffusion model then generates the final output image based on the intermediate output image. The final output image includes inpainted content in the second region that is consistent with the content in the first region.

Image and object inpainting with diffusion models

The present disclosure relates to systems, methods, and non-transitory computer readable media that generate inpainted digital images utilizing class-specific cascaded modulation inpainting neural network. For example, the disclosed systems utilize a class-specific cascaded modulation inpainting neural network that includes cascaded modulation decoder layers to generate replacement pixels portraying a particular target object class. To illustrate, in response to user selection of a replacement region and target object class, the disclosed systems utilize a class-specific cascaded modulation inpainting neural network corresponding to the target object class to generate an inpainted digital image that portrays an instance of the target object class within the replacement region. Moreover, in one or more embodiments the disclosed systems train class-specific cascaded modulation inpainting neural networks corresponding to a variety of target object classes, such as a sky object class, a water object class, a ground object class, or a human object class.

Object class inpainting in digital images utilizing class-specific inpainting neural networks

The present disclosure relates to systems, methods, and non-transitory computer-readable media that generates object shadows for digital images utilizing corresponding geometry-aware buffer channels. For instance, in one or more embodiments, the disclosed systems generate, utilizing a height prediction neural network, an object height map for a digital object portrayed in a digital image and a background height map for a background portrayed in the digital image. The disclosed systems also generate, from the digital image, a plurality of geometry-aware buffer channels using the object height map and the background height map. Further, the disclosed systems modify the digital image to include a soft object shadow for the digital object using the plurality of geometry-aware buffer channels.

Generating soft object shadows for general shadow receivers within digital images using geometry-aware buffer channels

Systems and methods for image segmentation are described. Embodiments of the present disclosure receive an image depicting an object; generate image features for the image by performing a convolutional self-attention operation that outputs a plurality of attention-weighted values for a convolutional kernel applied at a position of a sliding window on the image; and generate label data that identifies the object based on the image features.

Transformer for efficient image segmentation

The present disclosure relates to systems, methods, and non-transitory computer readable media for panoptically guiding digital image inpainting utilizing a panoptic inpainting neural network. In some embodiments, the disclosed systems utilize a panoptic inpainting neural network to generate an inpainted digital image according to panoptic segmentation map that defines pixel regions corresponding to different panoptic labels. In some cases, the disclosed systems train a neural network utilizing a semantic discriminator that facilitates generation of digital images that are realistic while also conforming to a semantic segmentation. The disclosed systems generate and provide a panoptic inpainting interface to facilitate user interaction for inpainting digital images. In certain embodiments, the disclosed systems iteratively update an inpainted digital image based on changes to a panoptic segmentation map.

Generating and providing a panoptic inpainting interface for generating and modifying inpainted digital images

Iteratively modifying inpainted digital images based on changes to panoptic segmentation maps

Learning parameters for neural networks using a semantic discriminator and an object-level discriminator

The present disclosure relates to systems, non-transitory computer-readable media, and methods for utilizing machine learning models to generate refined depth maps of digital images utilizing digital segmentation masks. In particular, in one or more embodiments, the disclosed systems generate a depth map for a digital image utilizing a depth estimation machine learning model, determine a digital segmentation mask for the digital image, and generate a refined depth map from the depth map and the digital segmentation mask utilizing a depth refinement machine learning model. In some embodiments, the disclosed systems generate first and second intermediate depth maps using the digital segmentation mask and an inverse digital segmentation mask and merger the first and second intermediate depth maps to generate the refined depth map.

Utilizing machine learning models to generate refined depth maps with segmentation mask guidance

This disclosure describes one or more implementations of a digital image semantic layout manipulation system that generates refined digital images resembling the style of one or more input images while following the structure of an edited semantic layout. For example, in various implementations, the digital image semantic layout manipulation system builds and utilizes a sparse attention warped image neural network to generate high-resolution warped images and a digital image layout neural network to enhance and refine the high-resolution warped digital image into a realistic and accurate refined digital image.

Generating digital images utilizing high-resolution sparse attention and semantic layout manipulation neural networks

The present disclosure relates to systems, non-transitory computer-readable media, and methods that utilize a progressive refinement network to refine alpha mattes generated utilizing a mask-guided matting neural network. In particular, the disclosed systems can use the matting neural network to process a digital image and a coarse guidance mask to generate alpha mattes at discrete neural network layers. In turn, the disclosed systems can use the progressive refinement network to combine alpha mattes and refine areas of uncertainty. For example, the progressive refinement network can combine a core alpha matte corresponding to more certain core regions of a first alpha matte and a boundary alpha matte corresponding to uncertain boundary regions of a second, higher resolution alpha matte. Based on the combination of the core alpha matte and the boundary alpha matte, the disclosed systems can generate a final alpha matte for use in image matting processes.

Generating alpha mattes utilizing deep learning

The present disclosure relates to systems, methods, and non-transitory computer-readable media that modify digital images via scene-based editing using image understanding facilitated by artificial intelligence. For example, in one or more embodiments the disclosed systems utilize generative machine learning models to create modified digital images portraying human subjects. In particular, the disclosed systems generate modified digital images by performing infill modifications to complete a digital image or human inpainting for portions of a digital image that portrays a human. Moreover, in some embodiments, the disclosed systems perform reposing of subjects portrayed within a digital image to generate modified digital images. In addition, the disclosed systems in some embodiments perform facial expression transfer and facial expression animations to generate modified digital images or animations.

Human inpainting utilizing a segmentation branch for generating an infill segmentation map

The present disclosure relates to systems, methods, and non-transitory computer-readable media that modify digital images via multi-layered scene completion techniques facilitated by artificial intelligence. For instance, in some embodiments, the disclosed systems receive a digital image portraying a first object and a second object against a background, where the first object occludes a portion of the second object. Additionally, the disclosed systems pre-process the digital image to generate a first content fill for the portion of the second object occluded by the first object and a second content fill for a portion of the background occluded by the second object. After pre-processing, the disclosed systems detect one or more user interactions to move or delete the first object from the digital image. The disclosed systems further modify the digital image by moving or deleting the first object and exposing the first content fill for the portion of the second object.

Modifying digital images via multi-layered scene completion facilitated by artificial intelligence

Systems and methods for image segmentation are described. Embodiments of the present disclosure receive an image depicting an object; generate image features for the image by performing an atrous self-attention operation based on a plurality of dilation rates for a convolutional kernel applied at a position of a sliding window on the image; and generate label data that identifies the object based on the image features.

Utilizing a generative machine learning model to create modified digital images from an infill semantic map

The present disclosure relates to systems, methods, and non-transitory computer readable media for accurately, efficiently, and flexibly generating harmonized digital images utilizing a self-supervised image harmonization neural network. In particular, the disclosed systems can implement, and learn parameters for, a self-supervised image harmonization neural network to extract content from one digital image (disentangled from its appearance) and appearance from another from another digital image (disentangled from its content). For example, the disclosed systems can utilize a dual data augmentation method to generate diverse triplets for parameter learning (including input digital images, reference digital images, and pseudo ground truth digital images), via cropping a digital image with perturbations using three-dimensional color lookup tables ('LUTs'). Additionally, the disclosed systems can utilize the self-supervised image harmonization neural network to generate harmonized digital images that depict content from one digital image having the appearance of another digital image.

Learning parameters for an image harmonization neural network to generate deep harmonized digital images

Growing community of inventors

Campbell, CA, United States of America

Jianming Zhang