Adobe, Inc.

York University

Other

Scott David Cohen

Brian Lynn Price

Zhe Lin

Zhihong Ding

Jianming Zhang

Ning Xu

Hailin Jin

Jonathan Worthen Brandt

Jingwan Lu

Stephen N Schiller

Mingyang Ling

Haitian Zheng

Gregg Darryl Wilensky

Elya Shechtman

Sohrab Amirghodsi

Xiaohui Shen

Jianchao Yang

Jen-Chan Jeff Chien

Walter Wei-Tuh Chang

Qing Liu

Khoi Pham

Gavin Stuart Peter Miller

Zhifei Zhang

Zhuoyuan Chen

Nathan Aaron Carr

Jimei Yang

Zeke Koch

Huixuan Tang

Yinan Zhao

Radomir Mech

Connelly Barnes

Luis Figueroa

Walter Wei Tuh Chang

Chenxi Zhang

Paul John Asente

Sarah Kong

Jason Wen Yong Kuen

Peng Wang

Yufei Wang

Jun Hao Liew

Luis Angel Figueroa

Zhe Zhu

Matthew Joss

Kevin Gary Smith

Walter Chang

Franck Dernoncourt

Vineet Batra

Ankit Phogat

Michael F Kraley

Jonathan Eisenmann

Bryan Russell

Martin E Newell

Stephen Diverdi

Long Mai

Kushal Kafle

Soo Ye Kim

Alper Ayvaci

Ejaz Ahmed

Bryan Stuart Morse

Sayan Ghosh

Joel A Howard

Jingnan Wang

Peng Zhou

Sijie Zhu

Darshan Prasad

Celso Gomes

Marco Forte

Dafang He

Yu-Wei Chao

Mohamed Hamdy Mahmoud Abdelbaky Elhoseiny

Gowri Somanath

Michael S Brown

Ryan A Rossi

Eunyee Koh

Trung Huu Bui

Sungchul Kim

Mark Andrew Ruzon

Vlad Ion Morariu

Alan Lee Erickson

Krishna Kumar Singh

Curtis Michael Wigington

Yuqian Zhou

Christopher Alan Tensmeyer

Zijun Wei

Su Chen

Mai Long

Quan Hung Tran

Qingxiong Yang

Klaas Stöeckmann

Lesley Ann Northam

Mahmoud Afifi

Charles Li Chen

Jie Feng

Thomas Muehrke

Darshan D Prasad

Jacqueline L Feild

Xiang Zhu

Ruotian Luo

Henghui Ding

Sangryul Jeon

The present disclosure relates to systems, methods, and non-transitory computer-readable media that implement perspective-aware object move operations for digital image editing. For instance, in some embodiments, the disclosed systems determine a vanishing point associated with a digital image portraying an object. Additionally, the disclosed systems detect one or more user interactions for moving the object within the digital image. Based on moving the object with respect to the vanishing point, the disclosed systems perform a perspective-based resizing of the object within the digital image.

Modifying digital images via perspective-aware object move

The present disclosure relates to systems, methods, and non-transitory computer readable media for panoptically guiding digital image inpainting utilizing a panoptic inpainting neural network. In some embodiments, the disclosed systems utilize a panoptic inpainting neural network to generate an inpainted digital image according to panoptic segmentation map that defines pixel regions corresponding to different panoptic labels. In some cases, the disclosed systems train a neural network utilizing a semantic discriminator that facilitates generation of digital images that are realistic while also conforming to a semantic segmentation. The disclosed systems generate and provide a panoptic inpainting interface to facilitate user interaction for inpainting digital images. In certain embodiments, the disclosed systems iteratively update an inpainted digital image based on changes to a panoptic segmentation map.

Panoptically guided inpainting utilizing a panoptic inpainting neural network

Systems, methods, and non-transitory computer-readable media embed a trained neural network within a digital image. For instance, in one or more embodiments, the systems identify out-of-gamut pixel values of a digital image in a first gamut, where the digital image is converted to the first gamut from a second gamut. Furthermore, the systems determine target pixel values of a target version of the digital image in the first gamut that correspond to the out-of-gamut pixel values. The systems train a neural network to predict the target pixel values in the first gamut based on the out-of-gamut pixel values. The systems embed the neural network within the digital image in the second gamut to allow for extraction of the embedded neural network from the digital image to restore the digital image to a larger gamut digital image.

Recovering gamut color loss utilizing lightweight neural networks

The present disclosure relates to systems, methods, and non-transitory computer-readable media that provides a graphical user interface experience to move objects and generate new shadows within a digital image scene. For instance, in one or more embodiments, the disclosed systems receive a digital image depicting a scene. The disclosed systems receive a selection to position an object in a first location within the scene. Further, the disclosed systems composite an image by placing the object at the first location within the scene of the digital image. Moreover, the disclosed systems generate a modified digital image having a shadow of the object where the shadow is consistent with the scene and provides the modified digital image to the client device.

Moving objects casting a shadow and generating proxy shadows within a digital image

The present disclosure relates to systems, methods, and non-transitory computer-readable media that modify digital images via scene-based editing using image understanding facilitated by artificial intelligence. For instance, in one or more embodiments, the disclosed systems receive a digital image from a client device. The disclosed systems detect, utilizing a shadow detection neural network, an object portrayed in the digital image. The disclosed systems detect, utilizing the shadow detection neural network, a shadow portrayed in the digital image. The disclosed systems generate, utilizing the shadow detection neural network, an object-shadow pair prediction that associates the shadow with the object.

Detecting shadows and corresponding objects in digital images

The present disclosure relates to systems, methods, and non-transitory computer-readable media that modify digital images via scene-based editing to remove a shadow for an object. For instance, in one or more embodiments, the disclosed systems receive a digital image depicting a scene. The disclosed systems access a shadow mask of the shadow in a first location. Further, the disclosed systems generate the modified digital image without the shadow by generating a fill for the first location that preserves a visible location of the first location. Moreover, the disclosed systems generate the digital image without the shadow for the object by combining the fill with the digital image.

Texture-preserving shadow removal in digital images utilizing generating inpainting models

Embodiments of the present invention provide systems, methods, and non-transitory computer storage media for parsing a given input referring expression into a parse structure and generating a semantic computation graph to identify semantic relationships among and between objects. At a high level, when embodiments of the preset invention receive a referring expression, a parse tree is created and mapped into a hierarchical subject, predicate, object graph structure that labeled noun objects in the referring expression, the attributes of the labeled noun objects, and predicate relationships (e.g., verb actions or spatial propositions) between the labeled objects. Embodiments of the present invention then transform the subject, predicate, object graph structure into a semantic computation graph that may be recursively traversed and interpreted to determine how noun objects, their attributes and modifiers, and interrelationships are provided to downstream image editing, searching, or caption indexing tasks.

Methods and systems for generating a semantic computation graph for understanding and grounding referring expressions

Systems and methods for image processing are described. Embodiments of the present disclosure receive an image comprising a first region that includes content and a second region to be inpainted. Noise is then added to the image to obtain a noisy image, and a plurality of intermediate output images are generated based on the noisy image using a diffusion model trained using a perceptual loss. The intermediate output images predict a final output image based on a corresponding intermediate noise level of the diffusion model. The diffusion model then generates the final output image based on the intermediate output image. The final output image includes inpainted content in the second region that is consistent with the content in the first region.

Image and object inpainting with diffusion models

The present disclosure relates to systems, methods, and non-transitory computer readable media that generate inpainted digital images utilizing class-specific cascaded modulation inpainting neural network. For example, the disclosed systems utilize a class-specific cascaded modulation inpainting neural network that includes cascaded modulation decoder layers to generate replacement pixels portraying a particular target object class. To illustrate, in response to user selection of a replacement region and target object class, the disclosed systems utilize a class-specific cascaded modulation inpainting neural network corresponding to the target object class to generate an inpainted digital image that portrays an instance of the target object class within the replacement region. Moreover, in one or more embodiments the disclosed systems train class-specific cascaded modulation inpainting neural networks corresponding to a variety of target object classes, such as a sky object class, a water object class, a ground object class, or a human object class.

Object class inpainting in digital images utilizing class-specific inpainting neural networks

The present disclosure relates to systems, methods, and non-transitory computer readable media that utilizes artificial intelligence to learn to recommend foreground object images for use in generating composite images based on geometry and/or lighting features. For instance, in one or more embodiments, the disclosed systems transform a foreground object image corresponding to a background image using at least one of a geometry transformation or a lighting transformation. The disclosed systems further generating predicted embeddings for the background image, the foreground object image, and the transformed foreground object image within a geometry-lighting-sensitive embedding space utilizing a geometry-lighting-aware neural network. Using a loss determined from the predicted embeddings, the disclosed systems update parameters of the geometry-lighting-aware neural network. The disclosed systems further provide a variety of efficient user interfaces for generating composite digital images.

Recommending objects for image composition using geometry-and-lighting aware search and efficient user interface workflows

The present disclosure relates to systems, methods, and non-transitory computer-readable media that modify digital images via scene-based editing using image understanding facilitated by artificial intelligence. For instance, in one or more embodiments, the disclosed systems provide, for display within a graphical user interface of a client device, a digital image displaying a plurality of objects, the plurality of objects comprising a plurality of different types of objects. The disclosed systems generate, utilizing a segmentation neural network and without user input, an object mask for objects of the plurality of objects. The disclosed systems determine, utilizing a distractor detection neural network, a classification for the objects of the plurality of objects. The disclosed systems remove at least one object from the digital image, based on classifying the at least one object as a distracting object, by deleting the object mask for the at least one object.

Removing distracting objects from digital images

Recommending objects for image composition using a geometry-and-lighting aware neural network

Generating and providing a panoptic inpainting interface for generating and modifying inpainted digital images

Iteratively modifying inpainted digital images based on changes to panoptic segmentation maps

Learning parameters for neural networks using a semantic discriminator and an object-level discriminator

Growing community of inventors

Cupertino, CA, United States of America

Scott David Cohen