Adobe, Inc.

Yulun Zhang

Zhaowen Wang

Zhe Lin

Jimei Yang

Chen Fang

Yilin Wang

Jose Ignacio Echevarria Vallespi

Zhifei Zhang

Stephen Diverdi

Systems and methods for texture hallucination with a large upscaling factor are described. Embodiments of the systems and methods may receive an input image and a reference image, extract an upscaled feature map from the input image, match the input image to a portion of the reference image, wherein a resolution of the reference image is higher than a resolution of the input image, concatenate the upscaled feature map with a reference feature map corresponding to the portion of the reference image to produce a concatenated feature map, and generate a reconstructed image based on the concatenated feature map using a machine learning model trained with a texture loss and a degradation loss, wherein the texture loss is based on a high frequency band filter, and the degradation loss is based on a downscaled version of the reconstructed image.

Texture hallucination for large-scale image super-resolution

A style of a digital image is transferred to another digital image of arbitrary resolution. A high-resolution (HR) content image is segmented into several low-resolution (LR) patches. The resolution of a style image is matched to have the same resolution as the LR content image patches. Style transfer is then performed on a patch-by-patch basis using, for example, a pair of feature transforms&#x2014;whitening and coloring. The patch-by-patch style transfer process is then repeated at several increasing resolutions, or scale levels, of both the content and style images. The results of the style transfer at each scale level are incorporated into successive scale levels up to and including the original HR scale. As a result, style transfer can be performed with images having arbitrary resolutions to produce visually pleasing results with good spatial consistency.

Hierarchical scale matching and patch estimation for image style transfer with arbitrary resolution

In implementations of transferring image style to content of a digital image, an image editing system includes an encoder that extracts features from a content image and features from a style image. A whitening and color transform generates coarse features from the content and style features extracted by the encoder for one pass of encoding and decoding. Hence, the processing delay and memory requirements are low. A feature transfer module iteratively transfers style features to the coarse feature map and generates a fine feature map. The image editing system fuses the fine features with the coarse features, and a decoder generates an output image with content of the content image in a style of the style image from the fused features. Accordingly, the image editing system efficiently transfers an image style to image content in real-time, without undesirable artifacts in the output image.

Transferring image style to content of a digital image

Growing community of inventors

Somerville, MA, United States of America

Yulun Zhang