Adobe, Inc.

Adobe Systems Inc.

Adobeinc.

Other

University of North Carolina at Chapel Hill

Zhe Lin

Xiaohui Shen

Jianming Zhang

Scott David Cohen

Jonathan Worthen Brandt

Radomir Mech

Xin Lu

Hailin Jin

Baldo Antonio Faieta

Jianchao Yang

Jimei Yang

Yilin Wang

Mingyang Ling

Saeid Motiian

Zhihong Ding

Elya Shechtman

Chen Fang

Kalyan Krishna Sunkavalli

Zhaowen Wang

Sohrab Amirghodsi

Ning Xu

He Zhang

Jingwan Lu

Connelly Barnes

Brian Lynn Price

Ajinkya Gorakhnath Kale

Jason Wen Yong Kuen

Zhifei Zhang

Haoxiang Li

Jen-Chan Jeff Chien

Trung Huu Bui

Handong Zhao

Federico Perazzi

Pranav Vineet Aggarwal

Zijun Wei

Qing Liu

Haitian Zheng

Franck Dernoncourt

Sarah Kong

Alex Charles Filipkowski

Jiahui Yu

Joon-Young Lee

Shabnam Ghadar

Chih-Yao Hsieh

Midhun Harikumar

Pramod Srinivasan

Zhuoyuan Chen

Yufei Wang

Khoi Pham

Oliver Wang

I-Ming Pao

Samarth Gulati

Walter Chang

Jose Ignacio Echevarria Vallespi

Jayant Kumar

Yijun Li

Ratheesh Kalarot

Peng Wang

Krishna Kumar Singh

Akhilesh Kumar

Su Chen

Vuong Le

Jue Wang

Byungmoon Kim

Gavin Stuart Peter Miller

Quan Anh Tran

Anant Gilra

Mai Long

Quan Hung Tran

Brett Michael Butterfield

Matthew David Fisher

Lubomir D Bourdev

Richard Zhang

Duygu Ceylan Aksit

Zhou Ren

Bernard Kerr

Jonathan Eisenmann

Michael Kaplan

John Philip Collomosse

Yuyan Song

Zhixin Shu

Simon Chen

Michael A Marth

Long Mai

William L Marino

Yangtuanfeng Wang

Sheng Li

Fabian David Caba Heilbron

Daniel Miranda

Yinglan Ma

Kevin Gary Smith

Walter Wei-Tuh Chang

Yang Yang

Wei-An Lin

Simon Niklaus

Jiuxiang Gu

Yuqian Zhou

Alexei A Efros

Hao Hu

Florian Kainz

The present disclosure relates to systems, methods, and non-transitory computer readable media that generate inpainted digital images utilizing class-specific cascaded modulation inpainting neural network. For example, the disclosed systems utilize a class-specific cascaded modulation inpainting neural network that includes cascaded modulation decoder layers to generate replacement pixels portraying a particular target object class. To illustrate, in response to user selection of a replacement region and target object class, the disclosed systems utilize a class-specific cascaded modulation inpainting neural network corresponding to the target object class to generate an inpainted digital image that portrays an instance of the target object class within the replacement region. Moreover, in one or more embodiments the disclosed systems train class-specific cascaded modulation inpainting neural networks corresponding to a variety of target object classes, such as a sky object class, a water object class, a ground object class, or a human object class.

Object class inpainting in digital images utilizing class-specific inpainting neural networks

One or more aspects of a method, apparatus, and non-transitory computer readable medium include obtaining an input description and an input image depicting a subject, encoding the input description using a text encoder of an image generation model to obtain a text embedding, and encoding the input image using a subject encoder of the image generation model to obtain a subject embedding. A guidance embedding is generated by combining the subject embedding and the text embedding, and then an output image is generated based on the guidance embedding using a diffusion model of the image generation model. The output image depicts aspects of the subject and the input description.

Personalized text-to-image generation

Systems and methods for natural language processing are described. Embodiments are configured to receive a structured representation of a search query, wherein the structured representation comprises a plurality of nodes and at least one edge connecting two of the nodes, receive a modification expression for the search query, wherein the modification expression comprises a natural language expression, generate a modified structured representation based on the structured representation and the modification expression using a neural network configured to combine structured representation features and natural language expression features, and perform a search based on the modified structured representation.

Scene graph modification based on natural language commands

The present disclosure relates to systems, methods, and non-transitory computer readable media that utilizes artificial intelligence to learn to recommend foreground object images for use in generating composite images based on geometry and/or lighting features. For instance, in one or more embodiments, the disclosed systems transform a foreground object image corresponding to a background image using at least one of a geometry transformation or a lighting transformation. The disclosed systems further generating predicted embeddings for the background image, the foreground object image, and the transformed foreground object image within a geometry-lighting-sensitive embedding space utilizing a geometry-lighting-aware neural network. Using a loss determined from the predicted embeddings, the disclosed systems update parameters of the geometry-lighting-aware neural network. The disclosed systems further provide a variety of efficient user interfaces for generating composite digital images.

Recommending objects for image composition using geometry-and-lighting aware search and efficient user interface workflows

The present disclosure relates to systems, methods, and non-transitory computer-readable media that modify digital images via scene-based editing using image understanding facilitated by artificial intelligence. For instance, in one or more embodiments, the disclosed systems provide, for display within a graphical user interface of a client device, a digital image displaying a plurality of objects, the plurality of objects comprising a plurality of different types of objects. The disclosed systems generate, utilizing a segmentation neural network and without user input, an object mask for objects of the plurality of objects. The disclosed systems determine, utilizing a distractor detection neural network, a classification for the objects of the plurality of objects. The disclosed systems remove at least one object from the digital image, based on classifying the at least one object as a distracting object, by deleting the object mask for the at least one object.

Removing distracting objects from digital images

Recommending objects for image composition using a geometry-and-lighting aware neural network

Systems and methods for image segmentation are described. Embodiments of the present disclosure receive an image depicting an object; generate image features for the image by performing a convolutional self-attention operation that outputs a plurality of attention-weighted values for a convolutional kernel applied at a position of a sliding window on the image; and generate label data that identifies the object based on the image features.

Transformer for efficient image segmentation

The present disclosure relates to systems, methods, and non-transitory computer readable media for panoptically guiding digital image inpainting utilizing a panoptic inpainting neural network. In some embodiments, the disclosed systems utilize a panoptic inpainting neural network to generate an inpainted digital image according to panoptic segmentation map that defines pixel regions corresponding to different panoptic labels. In some cases, the disclosed systems train a neural network utilizing a semantic discriminator that facilitates generation of digital images that are realistic while also conforming to a semantic segmentation. The disclosed systems generate and provide a panoptic inpainting interface to facilitate user interaction for inpainting digital images. In certain embodiments, the disclosed systems iteratively update an inpainted digital image based on changes to a panoptic segmentation map.

Generating and providing a panoptic inpainting interface for generating and modifying inpainted digital images

Iteratively modifying inpainted digital images based on changes to panoptic segmentation maps

Learning parameters for neural networks using a semantic discriminator and an object-level discriminator

The present disclosure relates to systems, non-transitory computer-readable media, and methods for utilizing machine learning models to generate refined depth maps of digital images utilizing digital segmentation masks. In particular, in one or more embodiments, the disclosed systems generate a depth map for a digital image utilizing a depth estimation machine learning model, determine a digital segmentation mask for the digital image, and generate a refined depth map from the depth map and the digital segmentation mask utilizing a depth refinement machine learning model. In some embodiments, the disclosed systems generate first and second intermediate depth maps using the digital segmentation mask and an inverse digital segmentation mask and merger the first and second intermediate depth maps to generate the refined depth map.

Utilizing machine learning models to generate refined depth maps with segmentation mask guidance

This disclosure describes one or more implementations of a digital image semantic layout manipulation system that generates refined digital images resembling the style of one or more input images while following the structure of an edited semantic layout. For example, in various implementations, the digital image semantic layout manipulation system builds and utilizes a sparse attention warped image neural network to generate high-resolution warped images and a digital image layout neural network to enhance and refine the high-resolution warped digital image into a realistic and accurate refined digital image.

Generating digital images utilizing high-resolution sparse attention and semantic layout manipulation neural networks

Techniques for identity preserved controllable facial image manipulation are described that support generation of a manipulated digital image based on a facial image and a render image. For instance, a facial image depicting a facial representation of an individual is received as input. A feature space including an identity parameter and at least one other visual parameter is extracted from the facial image. An editing module edits one or more of the visual parameters and preserves the identity parameter. A renderer generates a render image depicting a morphable model reconstruction of the facial image based on the edit. The render image and facial image are encoded, and a generator of a neural network is implemented to generate a manipulated digital image based on the encoded facial image and the encoded render image.

Identity preserved controllable facial image manipulation

The present disclosure relates to systems, non-transitory computer-readable media, and methods that utilize a progressive refinement network to refine alpha mattes generated utilizing a mask-guided matting neural network. In particular, the disclosed systems can use the matting neural network to process a digital image and a coarse guidance mask to generate alpha mattes at discrete neural network layers. In turn, the disclosed systems can use the progressive refinement network to combine alpha mattes and refine areas of uncertainty. For example, the progressive refinement network can combine a core alpha matte corresponding to more certain core regions of a first alpha matte and a boundary alpha matte corresponding to uncertain boundary regions of a second, higher resolution alpha matte. Based on the combination of the core alpha matte and the boundary alpha matte, the disclosed systems can generate a final alpha matte for use in image matting processes.

Generating alpha mattes utilizing deep learning

Methods, systems, and non-transitory computer readable storage media are disclosed for generating semantic scene graphs for digital images using an external knowledgebase for feature refinement. For example, the disclosed system can determine object proposals and subgraph proposals for a digital image to indicate candidate relationships between objects in the digital image. The disclosed system can then extract relationships from an external knowledgebase for refining features of the object proposals and the subgraph proposals. Additionally, the disclosed system can generate a semantic scene graph for the digital image based on the refined features of the object/subgraph proposals. Furthermore, the disclosed system can update/train a semantic scene graph generation network based on the generated semantic scene graph. The disclosed system can also reconstruct the image using object labels based on the refined features to further update/train the semantic scene graph generation network.

Generating scene graphs from digital images using external knowledge and image reconstruction

Growing community of inventors

Fremont, CA, United States of America

Zhe Lin