Adobe, Inc.

Adobe Systems Inc.

Adobeinc.

Other

University of North Carolina at Chapel Hill

Zhe Lin

Xiaohui Shen

Jianming Zhang

Scott David Cohen

Jonathan Worthen Brandt

Radomir Mech

Xin Lu

Hailin Jin

Baldo Antonio Faieta

Jianchao Yang

Jimei Yang

Yilin Wang

Mingyang Ling

Elya Shechtman

Saeid Motiian

Zhihong Ding

Chen Fang

Kalyan Krishna Sunkavalli

Sohrab Amirghodsi

Zhaowen Wang

Jingwan Lu

Ning Xu

Connelly Barnes

He Zhang

Brian Lynn Price

Jason Wen Yong Kuen

Ajinkya Gorakhnath Kale

Zhifei Zhang

Haoxiang Li

Qing Liu

Haitian Zheng

Jen-Chan Jeff Chien

Franck Dernoncourt

Trung Huu Bui

Handong Zhao

Federico Perazzi

Pranav Vineet Aggarwal

Zijun Wei

Sarah Kong

Alex Charles Filipkowski

Jiahui Yu

Khoi Pham

Joon-Young Lee

Shabnam Ghadar

Chih-Yao Hsieh

Midhun Harikumar

Pramod Srinivasan

Zhuoyuan Chen

Yufei Wang

Walter Chang

Oliver Wang

I-Ming Pao

Samarth Gulati

Jose Ignacio Echevarria Vallespi

Jayant Kumar

Yijun Li

Ratheesh Kalarot

Peng Wang

Krishna Kumar Singh

Akhilesh Kumar

Su Chen

Vuong Le

Jue Wang

Byungmoon Kim

Gavin Stuart Peter Miller

Quan Anh Tran

Anant Gilra

Quan Hung Tran

Mai Long

Matthew David Fisher

Lubomir D Bourdev

Richard Zhang

Duygu Ceylan Aksit

Zhou Ren

Bernard Kerr

Jonathan Eisenmann

Michael Kaplan

John Philip Collomosse

Zhixin Shu

Patrick Reynolds

Yuyan Song

Simon Chen

Yuqian Zhou

Michael A Marth

William L Marino

Yangtuanfeng Wang

Long Mai

Sheng Li

Daniel Miranda

Fabian David Caba Heilbron

Yinglan Ma

Kevin Gary Smith

Walter Wei-Tuh Chang

Yang Yang

Wei-An Lin

Simon Niklaus

Jiuxiang Gu

Alexei A Efros

Hao Hu

Florian Kainz

Embodiments of the present invention provide systems, methods, and non-transitory computer storage media for parsing a given input referring expression into a parse structure and generating a semantic computation graph to identify semantic relationships among and between objects. At a high level, when embodiments of the preset invention receive a referring expression, a parse tree is created and mapped into a hierarchical subject, predicate, object graph structure that labeled noun objects in the referring expression, the attributes of the labeled noun objects, and predicate relationships (e.g., verb actions or spatial propositions) between the labeled objects. Embodiments of the present invention then transform the subject, predicate, object graph structure into a semantic computation graph that may be recursively traversed and interpreted to determine how noun objects, their attributes and modifiers, and interrelationships are provided to downstream image editing, searching, or caption indexing tasks.

Methods and systems for generating a semantic computation graph for understanding and grounding referring expressions

In implementations of systems for performing multiple segmentation tasks, a computing device implements a segment system to receive input data describing a digital image depicting an object. The segment system computes per-pixel embeddings for the digital image using a pixel decoder of a machine learning model. Output embeddings are generated using a transformer decoder of the machine learning model based on the per-pixel embeddings for the digital image, input embeddings for a first segmentation task and input embeddings for a second segmentation task. The segment system outputs a first digital image and a second digital image. The first digital image depicts the object segmented based on the first segmentation task and the second digital image depicts the object segmented based on the second segmentation task.

Performing multiple segmentation tasks

Systems and methods for image processing are described. Embodiments of the present disclosure receive an image comprising a first region that includes content and a second region to be inpainted. Noise is then added to the image to obtain a noisy image, and a plurality of intermediate output images are generated based on the noisy image using a diffusion model trained using a perceptual loss. The intermediate output images predict a final output image based on a corresponding intermediate noise level of the diffusion model. The diffusion model then generates the final output image based on the intermediate output image. The final output image includes inpainted content in the second region that is consistent with the content in the first region.

Image and object inpainting with diffusion models

Systems and methods for multi-modal image editing are provided. In one aspect, a system and method for multi-modal image editing includes identifying an image, a prompt identifying an element to be added to the image, and a mask indicating a first region of the image for depicting the element. The system then generates a partially noisy image map that includes noise in the first region and image features from the image in a second region outside the first region. A diffusion model generates a composite image map based on the partially noisy image map and the prompt. In some cases, the composite image map includes the target element in the first region that corresponds to the mask.

Multi-modal image editing

The present disclosure relates to systems, methods, and non-transitory computer readable media that generate inpainted digital images utilizing class-specific cascaded modulation inpainting neural network. For example, the disclosed systems utilize a class-specific cascaded modulation inpainting neural network that includes cascaded modulation decoder layers to generate replacement pixels portraying a particular target object class. To illustrate, in response to user selection of a replacement region and target object class, the disclosed systems utilize a class-specific cascaded modulation inpainting neural network corresponding to the target object class to generate an inpainted digital image that portrays an instance of the target object class within the replacement region. Moreover, in one or more embodiments the disclosed systems train class-specific cascaded modulation inpainting neural networks corresponding to a variety of target object classes, such as a sky object class, a water object class, a ground object class, or a human object class.

Object class inpainting in digital images utilizing class-specific inpainting neural networks

One or more aspects of a method, apparatus, and non-transitory computer readable medium include obtaining an input description and an input image depicting a subject, encoding the input description using a text encoder of an image generation model to obtain a text embedding, and encoding the input image using a subject encoder of the image generation model to obtain a subject embedding. A guidance embedding is generated by combining the subject embedding and the text embedding, and then an output image is generated based on the guidance embedding using a diffusion model of the image generation model. The output image depicts aspects of the subject and the input description.

Personalized text-to-image generation

Systems and methods for natural language processing are described. Embodiments are configured to receive a structured representation of a search query, wherein the structured representation comprises a plurality of nodes and at least one edge connecting two of the nodes, receive a modification expression for the search query, wherein the modification expression comprises a natural language expression, generate a modified structured representation based on the structured representation and the modification expression using a neural network configured to combine structured representation features and natural language expression features, and perform a search based on the modified structured representation.

Scene graph modification based on natural language commands

The present disclosure relates to systems, methods, and non-transitory computer readable media that utilizes artificial intelligence to learn to recommend foreground object images for use in generating composite images based on geometry and/or lighting features. For instance, in one or more embodiments, the disclosed systems transform a foreground object image corresponding to a background image using at least one of a geometry transformation or a lighting transformation. The disclosed systems further generating predicted embeddings for the background image, the foreground object image, and the transformed foreground object image within a geometry-lighting-sensitive embedding space utilizing a geometry-lighting-aware neural network. Using a loss determined from the predicted embeddings, the disclosed systems update parameters of the geometry-lighting-aware neural network. The disclosed systems further provide a variety of efficient user interfaces for generating composite digital images.

Recommending objects for image composition using geometry-and-lighting aware search and efficient user interface workflows

The present disclosure relates to systems, methods, and non-transitory computer-readable media that modify digital images via scene-based editing using image understanding facilitated by artificial intelligence. For instance, in one or more embodiments, the disclosed systems provide, for display within a graphical user interface of a client device, a digital image displaying a plurality of objects, the plurality of objects comprising a plurality of different types of objects. The disclosed systems generate, utilizing a segmentation neural network and without user input, an object mask for objects of the plurality of objects. The disclosed systems determine, utilizing a distractor detection neural network, a classification for the objects of the plurality of objects. The disclosed systems remove at least one object from the digital image, based on classifying the at least one object as a distracting object, by deleting the object mask for the at least one object.

Removing distracting objects from digital images

Recommending objects for image composition using a geometry-and-lighting aware neural network

Systems and methods for image segmentation are described. Embodiments of the present disclosure receive an image depicting an object; generate image features for the image by performing a convolutional self-attention operation that outputs a plurality of attention-weighted values for a convolutional kernel applied at a position of a sliding window on the image; and generate label data that identifies the object based on the image features.

Transformer for efficient image segmentation

The present disclosure relates to systems, methods, and non-transitory computer readable media for panoptically guiding digital image inpainting utilizing a panoptic inpainting neural network. In some embodiments, the disclosed systems utilize a panoptic inpainting neural network to generate an inpainted digital image according to panoptic segmentation map that defines pixel regions corresponding to different panoptic labels. In some cases, the disclosed systems train a neural network utilizing a semantic discriminator that facilitates generation of digital images that are realistic while also conforming to a semantic segmentation. The disclosed systems generate and provide a panoptic inpainting interface to facilitate user interaction for inpainting digital images. In certain embodiments, the disclosed systems iteratively update an inpainted digital image based on changes to a panoptic segmentation map.

Generating and providing a panoptic inpainting interface for generating and modifying inpainted digital images

Iteratively modifying inpainted digital images based on changes to panoptic segmentation maps

Learning parameters for neural networks using a semantic discriminator and an object-level discriminator

The present disclosure relates to systems, non-transitory computer-readable media, and methods for utilizing machine learning models to generate refined depth maps of digital images utilizing digital segmentation masks. In particular, in one or more embodiments, the disclosed systems generate a depth map for a digital image utilizing a depth estimation machine learning model, determine a digital segmentation mask for the digital image, and generate a refined depth map from the depth map and the digital segmentation mask utilizing a depth refinement machine learning model. In some embodiments, the disclosed systems generate first and second intermediate depth maps using the digital segmentation mask and an inverse digital segmentation mask and merger the first and second intermediate depth maps to generate the refined depth map.

Utilizing machine learning models to generate refined depth maps with segmentation mask guidance

Growing community of inventors

Fremont, CA, United States of America

Zhe Lin