Adobe, Inc.

Jiahui Yu

Zhe Lin

Xiaohui Shen

Jimei Yang

Xin Lu

Predicting patch displacement maps using a neural network is described. Initially, a digital image on which an image editing operation is to be performed is provided as input to a patch matcher having an offset prediction neural network. From this image and based on the image editing operation for which this network is trained, the offset prediction neural network generates an offset prediction formed as a displacement map, which has offset vectors that represent a displacement of pixels of the digital image to different locations for performing the image editing operation. Pixel values of the digital image are copied to the image pixels affected by the operation.

Predicting patch displacement maps using a neural network

Digital image completion by learning generation and patch matching jointly is described. Initially, a digital image having at least one hole is received. This holey digital image is provided as input to an image completer formed with a dual-stage framework that combines a coarse image neural network and an image refinement network. The coarse image neural network generates a coarse prediction of imagery for filling the holes of the holey digital image. The image refinement network receives the coarse prediction as input, refines the coarse prediction, and outputs a filled digital image having refined imagery that fills these holes. The image refinement network generates refined imagery using a patch matching technique, which includes leveraging information corresponding to patches of known pixels for filtering patches generated based on the coarse prediction. Based on this, the image completer outputs the filled digital image with the refined imagery.

Digital image completion by learning generation and patch matching jointly

Digital image completion using deep learning is described. Initially, a digital image having at least one hole is received. This holey digital image is provided as input to an image completer formed with a framework that combines generative and discriminative neural networks based on learning architecture of the generative adversarial networks. From the holey digital image, the generative neural network generates a filled digital image having hole-filling content in place of holes. The discriminative neural networks detect whether the filled digital image and the hole-filling digital content correspond to or include computer-generated content or are photo-realistic. The generating and detecting are iteratively continued until the discriminative neural networks fail to detect computer-generated content for the filled digital image and hole-filling content or until detection surpasses a threshold difficulty. Responsive to this, the image completer outputs the filled digital image with hole-filling content in place of the holey digital image's holes.

Digital image completion using deep learning

Certain embodiments involve using an image completion neural network to perform user-guided image completion. For example, an image editing application accesses an input image having a completion region to be replaced with new image content. The image editing application also receives a guidance input that is applied to a portion of a completion region. The image editing application provides the input image and the guidance input to an image completion neural network that is trained to perform image-completion operations using guidance input. The image editing application produces a modified image by replacing the completion region of the input image with the new image content generated with the image completion network. The image editing application outputs the modified image having the new image content.

User-guided image completion with image completion neural networks

Predicting patch displacement maps using a neural network is described. Initially, a digital image on which an image editing operation is to be performed is provided as input to a patch matcher having an offset prediction neural network. From this image and based on the image editing operation for which this network is trained, the offset prediction neural network generates an offset prediction formed as a displacement map, which has offset vectors that represent a displacement of pixels of the digital image to different locations for performing the image editing operation. Pixel values of the digital image are copied to the image pixels affected by the operation by: determining the vectors pixels that correspond to the image pixels affected by the image editing operation and mapping the pixel values of the image pixels represented by the determined offset vectors to the affected pixels. According to this mapping, the pixel values of the affected pixels are set, effective to perform the image editing operation.

Growing community of inventors

Champaign, IL, United States of America

Jiahui Yu