Adobe, Inc.

Adobe Systems Inc.

Adobeinc.

Other

York University

Scott David Cohen

Brian Lynn Price

Zhe Lin

Ning Xu

Zhihong Ding

Hailin Jin

Jianming Zhang

Stephen N Schiller

Mingyang Ling

Jonathan Worthen Brandt

Gregg Darryl Wilensky

Jingwan Lu

Haitian Zheng

Elya Shechtman

Xiaohui Shen

Jen-Chan Jeff Chien

Jianchao Yang

Sohrab Amirghodsi

Walter Wei-Tuh Chang

Gavin Stuart Peter Miller

Zhifei Zhang

Qing Liu

Zhuoyuan Chen

Khoi Pham

Nathan Aaron Carr

Jimei Yang

Zeke Koch

Huixuan Tang

Yinan Zhao

Radomir Mech

Connelly Barnes

Walter Wei Tuh Chang

Chenxi Zhang

Paul John Asente

Sarah Kong

Peng Wang

Jason Wen Yong Kuen

Yufei Wang

Jun Hao Liew

Luis Angel Figueroa

Zhe Zhu

Kevin Gary Smith

Vineet Batra

Ankit Phogat

Jonathan Eisenmann

Michael F Kraley

Bryan Russell

Martin E Newell

Stephen Diverdi

Long Mai

Kushal Kafle

Alper Ayvaci

Ejaz Ahmed

Bryan Stuart Morse

Sayan Ghosh

Joel A Howard

Jingnan Wang

Peng Zhou

Sijie Zhu

Marco Forte

Matthew Joss

Dafang He

Yu-Wei Chao

Mohamed Hamdy Mahmoud Abdelbaky Elhoseiny

Gowri Somanath

Michael S Brown

Walter Chang

Franck Dernoncourt

Ryan A Rossi

Eunyee Koh

Trung Huu Bui

Sungchul Kim

Mark Andrew Ruzon

Vlad Ion Morariu

Alan Lee Erickson

Curtis Michael Wigington

Christopher Alan Tensmeyer

Krishna Kumar Singh

Zijun Wei

Yuqian Zhou

Su Chen

Mai Long

Quan Hung Tran

Qingxiong Yang

Lesley Ann Northam

Mahmoud Afifi

Charles Li Chen

Jie Feng

Klaas Stoeckmann

Thomas Muehrke

Darshan D Prasad

Jacqueline L Feild

Xiang Zhu

Ruotian Luo

Henghui Ding

Sangryul Jeon

David Hart

Chenyun Wu

Daniil Pakhomov

Gowri Sornanath

Systems and methods for image processing are described. Embodiments of the present disclosure receive an image comprising a first region that includes content and a second region to be inpainted. Noise is then added to the image to obtain a noisy image, and a plurality of intermediate output images are generated based on the noisy image using a diffusion model trained using a perceptual loss. The intermediate output images predict a final output image based on a corresponding intermediate noise level of the diffusion model. The diffusion model then generates the final output image based on the intermediate output image. The final output image includes inpainted content in the second region that is consistent with the content in the first region.

Image and object inpainting with diffusion models

The present disclosure relates to systems, methods, and non-transitory computer readable media that generate inpainted digital images utilizing class-specific cascaded modulation inpainting neural network. For example, the disclosed systems utilize a class-specific cascaded modulation inpainting neural network that includes cascaded modulation decoder layers to generate replacement pixels portraying a particular target object class. To illustrate, in response to user selection of a replacement region and target object class, the disclosed systems utilize a class-specific cascaded modulation inpainting neural network corresponding to the target object class to generate an inpainted digital image that portrays an instance of the target object class within the replacement region. Moreover, in one or more embodiments the disclosed systems train class-specific cascaded modulation inpainting neural networks corresponding to a variety of target object classes, such as a sky object class, a water object class, a ground object class, or a human object class.

Object class inpainting in digital images utilizing class-specific inpainting neural networks

The present disclosure relates to systems, methods, and non-transitory computer readable media that utilizes artificial intelligence to learn to recommend foreground object images for use in generating composite images based on geometry and/or lighting features. For instance, in one or more embodiments, the disclosed systems transform a foreground object image corresponding to a background image using at least one of a geometry transformation or a lighting transformation. The disclosed systems further generating predicted embeddings for the background image, the foreground object image, and the transformed foreground object image within a geometry-lighting-sensitive embedding space utilizing a geometry-lighting-aware neural network. Using a loss determined from the predicted embeddings, the disclosed systems update parameters of the geometry-lighting-aware neural network. The disclosed systems further provide a variety of efficient user interfaces for generating composite digital images.

Recommending objects for image composition using geometry-and-lighting aware search and efficient user interface workflows

The present disclosure relates to systems, methods, and non-transitory computer-readable media that modify digital images via scene-based editing using image understanding facilitated by artificial intelligence. For instance, in one or more embodiments, the disclosed systems provide, for display within a graphical user interface of a client device, a digital image displaying a plurality of objects, the plurality of objects comprising a plurality of different types of objects. The disclosed systems generate, utilizing a segmentation neural network and without user input, an object mask for objects of the plurality of objects. The disclosed systems determine, utilizing a distractor detection neural network, a classification for the objects of the plurality of objects. The disclosed systems remove at least one object from the digital image, based on classifying the at least one object as a distracting object, by deleting the object mask for the at least one object.

Removing distracting objects from digital images

Recommending objects for image composition using a geometry-and-lighting aware neural network

The present disclosure relates to systems, methods, and non-transitory computer readable media for panoptically guiding digital image inpainting utilizing a panoptic inpainting neural network. In some embodiments, the disclosed systems utilize a panoptic inpainting neural network to generate an inpainted digital image according to panoptic segmentation map that defines pixel regions corresponding to different panoptic labels. In some cases, the disclosed systems train a neural network utilizing a semantic discriminator that facilitates generation of digital images that are realistic while also conforming to a semantic segmentation. The disclosed systems generate and provide a panoptic inpainting interface to facilitate user interaction for inpainting digital images. In certain embodiments, the disclosed systems iteratively update an inpainted digital image based on changes to a panoptic segmentation map.

Generating and providing a panoptic inpainting interface for generating and modifying inpainted digital images

Iteratively modifying inpainted digital images based on changes to panoptic segmentation maps

Learning parameters for neural networks using a semantic discriminator and an object-level discriminator

This disclosure describes one or more implementations of a digital image semantic layout manipulation system that generates refined digital images resembling the style of one or more input images while following the structure of an edited semantic layout. For example, in various implementations, the digital image semantic layout manipulation system builds and utilizes a sparse attention warped image neural network to generate high-resolution warped images and a digital image layout neural network to enhance and refine the high-resolution warped digital image into a realistic and accurate refined digital image.

Generating digital images utilizing high-resolution sparse attention and semantic layout manipulation neural networks

The present disclosure relates to systems, methods, and non-transitory computer-readable media that modify digital images via multi-layered scene completion techniques facilitated by artificial intelligence. For instance, in some embodiments, the disclosed systems receive a digital image portraying a first object and a second object against a background, where the first object occludes a portion of the second object. Additionally, the disclosed systems pre-process the digital image to generate a first content fill for the portion of the second object occluded by the first object and a second content fill for a portion of the background occluded by the second object. After pre-processing, the disclosed systems detect one or more user interactions to move or delete the first object from the digital image. The disclosed systems further modify the digital image by moving or deleting the first object and exposing the first content fill for the portion of the second object.

Modifying digital images via multi-layered scene completion facilitated by artificial intelligence

The present disclosure relates to systems, methods, and non-transitory computer-readable media that modify digital images using an intelligent user interface tool that determines the intent of a user interaction. For instance, in some embodiments, the disclosed systems receive, via a graphical user interface of a client device, a user interaction with a set of pixels within a digital image. The disclosed systems determine, based on the user interaction, a user intent for targeting one or more portions of the digital image for deletion, the one or more portions including an additional set of pixels that differs from the set of pixels. Based on the user intent, the disclosed systems modify the digital image to delete the one or more portions from the digital image.

Modifying digital images utilizing intent deterministic user interface tools

The present disclosure relates to systems, methods, and non-transitory computer-readable media that modify digital images via scene-based editing using image understanding facilitated by artificial intelligence. For instance, in one or more embodiments, the disclosed systems generate utilizing a segmentation neural network, an object mask for each object of a plurality of objects of a digital image. The disclosed systems detect a first user interaction with an object in the digital image displayed via a graphical user interface. The disclosed systems surface, via the graphical user interface, the object mask for the object in response to the first user interaction. The disclosed systems perform an object-aware modification of the digital image in response to a second user interaction with the object mask for the object.

Modifying digital images via scene-based editing using image understanding facilitated by artificial intelligence

The present disclosure relates to systems, methods, and non-transitory computer-readable media that generates object masks for digital objects portrayed in digital images utilizing a detection-masking neural network pipeline. In particular, in one or more embodiments, the disclosed systems utilize detection heads of a neural network to detect digital objects portrayed within a digital image. In some cases, each detection head is associated with one or more digital object classes that are not associated with the other detection heads. Further, in some cases, the detection heads implement multi-scale synchronized batch normalization to normalize feature maps across various feature levels. The disclosed systems further utilize a masking head of the neural network to generate one or more object masks for the detected digital objects. In some cases, the disclosed systems utilize post-processing techniques to filter out low-quality masks.

Detecting digital objects and generating object masks on device

The present disclosure relates to systems, methods, and non-transitory computer-readable media that implements related image search and image modification processes using various search engines and a consolidated graphical user interface. For instance, in one or more embodiments, the disclosed systems receive an input digital image and search input and further modify the input digital image using the image search results retrieved in response to the search input. In some cases, the search input includes a multi-modal search input having multiple queries (e.g., an image query and a text query), and the disclosed systems retrieve the image search results utilizing a weighted combination of the queries. In some implementations, the disclosed systems generate an input embedding for the search input (e.g., the multi-modal search input) and retrieve the image search results using the input embedding.

Generating unified embeddings from multi-modal canvas inputs for image retrieval

The present disclosure relates to systems, non-transitory computer-readable media, and methods for training and utilizing scale-diverse segmentation neural networks to analyze digital images at different scales and identify different target objects portrayed in the digital images. For example, in one or more embodiments, the disclosed systems analyze a digital image and corresponding user indicators (e.g., foreground indicators, background indicators, edge indicators, boundary region indicators, and/or voice indicators) at different scales utilizing a scale-diverse segmentation neural network. In particular, the disclosed systems can utilize the scale-diverse segmentation neural network to generate a plurality of semantically meaningful object segmentation outputs. Furthermore, the disclosed systems can provide the plurality of object segmentation outputs for display and selection to improve the efficiency and accuracy of identifying target objects and modifying the digital image.

Segmenting objects using scale-diverse segmentation neural networks

Growing community of inventors

Cupertino, CA, United States of America

Scott David Cohen