Adobe, Inc.

Microsoft Technology Licensing, LLC

Yuqian Zhou

Elya Shechtman

Sohrab Amirghodsi

Connelly Barnes

Timothy Andrew Large

Neil Emerton

Yonghuan David Ren

Zhe Lin

Se Hoon Lim

Scott David Cohen

Jianming Zhang

Jingwan Lu

Zijun Wei

Qing Liu

Haitian Zheng

Florian Kainz

Lingzhi Zhang

Xuaner Zhang

Yunhan Zhao

Mang Tik Chiu

Methods, systems, and non-transitory computer readable storage media are disclosed for generating neural network based perceptual artifact segmentations in synthetic digital image content. The disclosed system utilizing neural networks to detect perceptual artifacts in digital images in connection with generating or modifying digital images. The disclosed system determines a digital image including one or more synthetically modified portions. The disclosed system utilizes an artifact segmentation machine-learning model to detect perceptual artifacts in the synthetically modified portion(s). The artifact segmentation machine-learning model is trained to detect perceptual artifacts based on labeled artifact regions of synthetic training digital images. Additionally, the disclosed system utilizes the artifact segmentation machine-learning model in an iterative inpainting process. The disclosed system utilizes one or more digital image inpainting models to inpaint in a digital image. The disclosed system utilizes the artifact segmentation machine-learning model detect perceptual artifacts in the inpainted portions for additional inpainting iterations.

Generating iterative inpainting digital images via neural network based perceptual artifact segmentations

The present disclosure relates to systems, methods, and non-transitory computer readable media for accurately generating inpainted digital images utilizing a guided inpainting model guided by both plane panoptic segmentation and plane grouping. For example, the disclosed systems utilize a guided inpainting model to fill holes of missing pixels of a digital image as informed or guided by an appearance guide and a geometric guide. Specifically, the disclosed systems generate an appearance guide utilizing plane panoptic segmentation and generate a geometric guide by grouping plane panoptic segments. In some embodiments, the disclosed systems generate a modified digital image by implementing an inpainting model guided by both the appearance guide (e.g., a plane panoptic segmentation map) and the geometric guide (e.g., a plane grouping map).

Improving digital image inpainting utilizing plane panoptic segmentation and plane grouping

Systems and methods for image processing are described. Embodiments of the present disclosure receive an image comprising a first region that includes content and a second region to be inpainted. Noise is then added to the image to obtain a noisy image, and a plurality of intermediate output images are generated based on the noisy image using a diffusion model trained using a perceptual loss. The intermediate output images predict a final output image based on a corresponding intermediate noise level of the diffusion model. The diffusion model then generates the final output image based on the intermediate output image. The final output image includes inpainted content in the second region that is consistent with the content in the first region.

Image and object inpainting with diffusion models

Embodiments are disclosed for performing wire segmentation of images using machine learning. In particular, in one or more embodiments, the disclosed systems and methods comprise receiving an input image, generating, by a first trained neural network model, a global probability map representation of the input image indicating a probability value of each pixel including a representation of wires, and identifying regions of the input image indicated as including the representation of wires. The disclosed systems and methods further comprise, for each region from the identified regions, concatenating the region and information from the global probability map to create a concatenated input, and generating, by a second trained neural network model, a local probability map representation of the region based on the concatenated input, indicating pixels of the region including representations of wires. The disclosed systems and methods further comprise aggregating local probability maps for each region.

Wire segmentation for images using machine learning

Examples are disclosed that relate to the restoration of degraded images acquired via a behind-display camera. One example provides a method of training a machine learning model, the method comprising inputting training image pairs into the machine learning model, each training image pair comprising an undegraded image and a degraded image that represents an appearance of the undegraded image to a behind-display camera, and training the machine learning model using the training image pairs to generate frequency information that is missing from the degraded images.

Image restoration for through-display imaging

An electronic device comprises a display, an illumination source, a camera, and a logic system. The illumination source is configured to project structured illumination onto a subject. The camera is configured to image the subject through the display, which includes collecting the structured illumination as reflected by the subject. The logic system is configured to receive, from the camera, a digital image of the subject imaged through the display. The logic system is further configured to sharpen the digital image based on the spatially resolved intensity of the structured illumination as reflected by the subject.

Image sharpening for subjects imaged through display

The present disclosure relates to systems, methods, and non-transitory computer-readable media for accurately restoring missing pixels within a hole region of a target image utilizing multi-image inpainting techniques based on incorporating geometric depth information. For example, in various implementations, the disclosed systems utilize a depth prediction of a source image as well as camera relative pose parameters. Additionally, in some implementations, the disclosed systems jointly optimize the depth rescaling and camera pose parameters before generating the reprojected image to further increase the accuracy of the reprojected image. Further, in various implementations, the disclosed systems utilize the reprojected image in connection with a content-aware fill model to generate a refined composite image that includes the target image having a hole, where the hole is filled in based on the reprojected image of the source image.

Image reprojection and multi-image inpainting based on geometric depth parameters

Digital image inpainting utilizing plane panoptic segmentation and plane grouping

Various disclosed embodiments are directed to inpainting one or more portions of a target image based on merging (or selecting) one or more portions of a warped image with (or from) one or more portions of an inpainting candidate (e.g., via a learning model). This, among other functionality described herein, resolves the inaccuracies of existing image inpainting technologies.

Growing community of inventors

Urbana, IL, United States of America

Yuqian Zhou