Adobe, Inc.

Memsic Transducer Systems Co., Ltd.

Telefonaktiebolaget Lm Ericsson (Publ)

Beijing Jingdong Shangke Information Technology Co., Ltd.

Jd.com American Technologies Corporation

Memsic, Inc.

Ning Xu

Scott David Cohen

Zhe Lin

Jianming Zhang

Brian Lynn Price

Jingwan Lu

Haitian Zheng

Hailin Jin

Elya Shechtman

Sohrab Amirghodsi

Zhaowen Wang

Franck Dernoncourt

Qing Liu

Alan Broad

Tianlang Chen

Martin Albert Turon

Connelly Barnes

Rahul Kapur

Jing Shi

Jaidev Prabhu

Trung Huu Bui

Stephen N Schiller

Yilin Wang

Joon-Young Lee

He Zhang

Seokhwan Kim

Wentian Zhao

Xin Yang

Seoungwug Oh

Matt Miller

Marco Forte

Qihang Yu

Radomir Mech

Jonathan Worthen Brandt

Jimei Yang

Byungmoon Kim

Prashant Anand

Ramanathan Lakshmikanthan

Hui Zhou

John Philip Collomosse

Amir Pouran Ben Veyseh

Mai Long

Li Chen

Shizhu Liu

Sun Den Chen

Daicho Ito

Biao Jia

Naoto Inoue

Jianxun Sun

Kailin Huang

Bayram Safa Cicek

Chengchong Zhang

Haotian Zhang

The present disclosure relates to systems, methods, and non-transitory computer readable media for panoptically guiding digital image inpainting utilizing a panoptic inpainting neural network. In some embodiments, the disclosed systems utilize a panoptic inpainting neural network to generate an inpainted digital image according to panoptic segmentation map that defines pixel regions corresponding to different panoptic labels. In some cases, the disclosed systems train a neural network utilizing a semantic discriminator that facilitates generation of digital images that are realistic while also conforming to a semantic segmentation. The disclosed systems generate and provide a panoptic inpainting interface to facilitate user interaction for inpainting digital images. In certain embodiments, the disclosed systems iteratively update an inpainted digital image based on changes to a panoptic segmentation map.

Panoptically guided inpainting utilizing a panoptic inpainting neural network

The present disclosure relates to systems, methods, and non-transitory computer readable media that generate inpainted digital images utilizing class-specific cascaded modulation inpainting neural network. For example, the disclosed systems utilize a class-specific cascaded modulation inpainting neural network that includes cascaded modulation decoder layers to generate replacement pixels portraying a particular target object class. To illustrate, in response to user selection of a replacement region and target object class, the disclosed systems utilize a class-specific cascaded modulation inpainting neural network corresponding to the target object class to generate an inpainted digital image that portrays an instance of the target object class within the replacement region. Moreover, in one or more embodiments the disclosed systems train class-specific cascaded modulation inpainting neural networks corresponding to a variety of target object classes, such as a sky object class, a water object class, a ground object class, or a human object class.

Object class inpainting in digital images utilizing class-specific inpainting neural networks

Generating and providing a panoptic inpainting interface for generating and modifying inpainted digital images

Iteratively modifying inpainted digital images based on changes to panoptic segmentation maps

Learning parameters for neural networks using a semantic discriminator and an object-level discriminator

This disclosure describes one or more implementations of a digital image semantic layout manipulation system that generates refined digital images resembling the style of one or more input images while following the structure of an edited semantic layout. For example, in various implementations, the digital image semantic layout manipulation system builds and utilizes a sparse attention warped image neural network to generate high-resolution warped images and a digital image layout neural network to enhance and refine the high-resolution warped digital image into a realistic and accurate refined digital image.

Generating digital images utilizing high-resolution sparse attention and semantic layout manipulation neural networks

The present disclosure relates to systems, non-transitory computer-readable media, and methods that utilize a progressive refinement network to refine alpha mattes generated utilizing a mask-guided matting neural network. In particular, the disclosed systems can use the matting neural network to process a digital image and a coarse guidance mask to generate alpha mattes at discrete neural network layers. In turn, the disclosed systems can use the progressive refinement network to combine alpha mattes and refine areas of uncertainty. For example, the progressive refinement network can combine a core alpha matte corresponding to more certain core regions of a first alpha matte and a boundary alpha matte corresponding to uncertain boundary regions of a second, higher resolution alpha matte. Based on the combination of the core alpha matte and the boundary alpha matte, the disclosed systems can generate a final alpha matte for use in image matting processes.

Generating alpha mattes utilizing deep learning

This disclosure describes one or more implementations of systems, non-transitory computer-readable media, and methods that perform language guided digital image editing utilizing a cycle-augmentation generative-adversarial neural network (CAGAN) that is augmented using a cross-modal cyclic mechanism. For example, the disclosed systems generate an editing description network that generates language embeddings which represent image transformations applied between a digital image and a modified digital image. The disclosed systems can further train a GAN to generate modified images by providing an input image and natural language embeddings generated by the editing description network (representing various modifications to the digital image from a ground truth modified image). In some instances, the disclosed systems also utilize an image request attention approach with the GAN to generate images that include adaptive edits in different spatial locations of the image.

Modifying digital images utilizing a language guided image editing model

The present disclosure relates to systems, methods, and non-transitory computer readable media for training a generative inpainting neural network to accurately generate inpainted digital images via object-aware training and/or masked regularization. For example, the disclosed systems utilize an object-aware training technique to learn parameters for a generative inpainting neural network based on masking individual object instances depicted within sample digital images of a training dataset. In some embodiments, the disclosed systems also (or alternatively) utilize a masked regularization technique as part of training to prevent overfitting by penalizing a discriminator neural network utilizing a regularization term that is based on an object mask. In certain cases, the disclosed systems further generate an inpainted digital image utilizing a trained generative inpainting model with parameters learned via the object-aware training and/or the masked regularization.

Learning parameters for generative inpainting neural networks utilizing object-aware training and masked regularization

The present disclosure relates to systems, methods, and non-transitory computer readable media that generate inpainted digital images utilizing a cascaded modulation inpainting neural network. For example, the disclosed systems utilize a cascaded modulation inpainting neural network that includes cascaded modulation decoder layers. For example, in one or more decoder layers, the disclosed systems start with global code modulation that captures the global-range image structures followed by an additional modulation that refines the global predictions. Accordingly, in one or more implementations, the image inpainting system provides a mechanism to correct distorted local details. Furthermore, in one or more implementations, the image inpainting system leverages fast Fourier convolutions block within different resolution layers of the encoder architecture to expand the receptive field of the encoder and to allow the network encoder to better capture global structure.

Digital image inpainting utilizing a cascaded modulation inpainting neural network

Systems and methods for image processing are described. Embodiments of the present disclosure identify a first image depicting a first object; identify a plurality of candidate images depicting a second object; select a second image from the plurality of candidate images depicting the second object based on the second image and a sequence of previous images including the first image using a crop selection network trained to select a next compatible image based on the sequence of previous images; and generate a composite image depicting the first object and the second object based on the first image and the second image.

Hierarchical image generation via transformer-based sequential patch selection

Systems and methods for natural language processing are described. One or more embodiments of the present disclosure generate a word representation vector for each word of a text comprising an event trigger word and an argument candidate word; generate a dependency tree based on the text and the word representation vector; determine that at least one word of the text is independent of a relationship between the event trigger word and the argument candidate word; remove the at least one word from the dependency tree based on the determination to obtain a pruned dependency tree; generate a modified representation vector for each word of the pruned dependency tree using a graph convolutional network (GCN); and identify the relationship between the event trigger word and the argument candidate word based on the modified representation vector for each word of the pruned dependency tree.

Event understanding with deep learning

The present disclosure relates to systems and methods for automatically processing images based on a user request. In some examples, a request is divided into a retouching command (e.g., a global edit) and an inpainting command (e.g., a local edit). A retouching mask and an inpainting mask are generated to indicate areas where the edits will be applied. A photo-request attention and a multi-modal modulation process are applied to features representing the image, and a modified image that incorporates the user's request is generated using the modified features.

Automatic photo editing via linguistic request

Certain aspects and features of this disclosure relate to graph-based video instance segmentation. In one example, a reference instance of an object in a reference frame and features in a target frame are identified and used to produce a graph of nodes and edges. Each node represents a feature in the target frame or the reference instance of the object in the reference frame. Each edge of the graph represents a spatiotemporal relationship between the feature in the target frame and the reference instance of the object. Embeddings of the nodes and edges of the graph are iteratively updated based on the spatiotemporal relationship between a feature in the target frame and the reference instance of the object in the reference frame, resulting in a fused node embedding that can be used for detecting the target instance of the object.

Graph-based video instance segmentation

This disclosure describes methods, non-transitory computer readable storage media, and systems that utilize weakly supervised graph matching to align an ungrounded label graph and a visual graph corresponding to a digital image. Specifically, the disclosed system utilizes a label embedding model to generate label graph embeddings from the ungrounded label graph and a visual embedding network to generate visual graph embeddings from the visual graph. Additionally, the disclosed system determines similarity metrics indicating the similarity of pairs of label graph embeddings and visual graph embeddings. The disclosed system then generates a semantic scene graph by utilizing a graph matching algorithm to align the ungrounded label graph and the visual graph based on the similarity metrics. In some embodiments, the disclosed system utilizes contrastive learning to modify the embedding models. Furthermore, in additional embodiments, the disclosed system utilizes the semantic scene graph to train a scene graph generation neural network.

Generating semantic scene graphs from ungrounded label graphs and visual graphs for digital images

Growing community of inventors

Milpitas, CA, United States of America

Ning Xu