Baidu USA LLC

Baidu.com Times Technology (Beijing) Co. Ltd.

Galileo Group, Inc.

Zhihong Pan

Baopu Li

Donald Michael Barnes

James Michael Grichnik

Yingze Bao

Daming Lu

Thorsten Mewes

Hsuchun Cheng

Xi Chelsea Chen

Gang Zhang

Dongliang He

Xi Chen

Wenhao Wu

Yanwen Fan

Teng Xi

Tianwei Lin

Image super-resolution (SR) refers to the process of recovering high-resolution (HR) images from low-resolution (LR) inputs. Blind image SR is a more challenging task which involves unknown blurring kernels and characterizes the degradation process from HR to LR. In the present disclosure, embodiments of a variational autoencoder (VAE) are leveraged to train a kernel autoencoder for more accurate degradation representation and more efficient kernel estimation. In one or more embodiments, a kernel-agnostic loss is used to learn more robust kernel features in the latent space from LR inputs without using ground-truth kernel references. In addition, attention-based adaptive pooling is introduced to improve kernel estimation accuracy, and spatially non-uniform kernel features are passed into SR restoration resulting in additional kernel estimation error tolerance. Extensive experiments on synthetic and real-world images show that embodiments of the presented model outperform state-of-the-art methods significantly with the peak signal-to-noise ratio (PSNR) raised considerably.

Robust and efficient blind super-resolution using variational kernel autoencoder

Network architecture search (NAS) received a lot of attention. The supernet-based differentiable approach is popular because it can effectively share the weights and lead to more efficient search. However, the mismatch between the architecture and weights caused by weight sharing still exists. Moreover, the coupling effects among different operators are also neglected. To alleviate these problems, embodiments of an effective NAS methodology by similarity-based operator ranking are presented herein. With the aim of approximating each layer's output in the supernet, a similarity-based operator ranking based on statistical random comparison is used. In one or more embodiments, then the operator that possibly causes the least change to feature distribution discrepancy is pruned. In one or more embodiments, a fair sampling process may be used to mitigate the operators' Matthew effect that happened frequently in previous supernet approaches.

Neural architecture search via similarity-based operator ranking

Presented herein are embodiments of systems and methods for training a system and for using a trained system to generate super-resolution imagery from low-resolution imagery. Embodiments for generating super-resolution imagery from low-resolution imagery include obtaining an input trade-off parameter that indicates a preference regarding low distortion or high perceptual quality for a generated SR image and obtaining a latent variable from a distribution defined, at least in part, by a trade-off parameter. Embodiments include inputting an LR image and the latent variable into an embodiment of an invertible rescaling network (IRNN) in an inverse upscaling direction of the IRNN to generate an output SR image that comprises accuracy and perception qualities conditioned by the input trade-off parameter. In one or more embodiments, a trained IRNN uses a trade-off parameter that indicates a desired trade-off between whether the trained IRNN generates an SR image having lower distortion or higher perceptual quality.

Systems and methods for adjustable image rescaling

According to one embodiment, a method is provided for video repainting performed by at least one processor of a computer device. The method includes receiving a video sequence having one or more image frames; detecting presences of a target object within the one or more image frames and determining pose condition and style shift of the detected objects; generating content representing a replacement object for the one or more image frames by applying the corresponding pose condition and style shift to the replacement object; and repainting the detected target object in the one or more image frames with the generated content.

System and process for repainting of planar objects in video

Presented herein are systems and methods for an end-to-end solution for object removal from a video using adaptive learning. In one or more embodiments, using a pre-trained image inpainting model, in-scene training data may be generated using optical-flow guided sampling. In one or more embodiments, the sampled patches are used to generate a training dataset, which is used to further train the image inpainting model until reaching a stop condition. The adaptively trained inpainting model may be used to generate a modified video in which the desired object or objects have been removed and the corresponding removed portion(s) have been filled (or inpainted) to preserve the integrity of the image.

Video content removal using flow-guided adaptive learning

Described herein are embodiments of a deep residual network dedicated to color filter array mosaic patterns. A mosaic stride convolution layer is introduced to match the mosaic pattern of a multispectral filter arrays (MSFA) or a color filter array raw image. Embodiments of a data augmentation using MSFA shifting and dynamic noise are applied to make the model robust to different noise levels. Embodiments of network optimization criteria may be created by using the noise standard deviation to normalize the Lloss function. Comprehensive experiments demonstrate that embodiments of the disclosed deep residual network outperform the state-of-the-art denoising algorithms in MSFA field.

Deep residual network for color filter array image denoising

Described herein are systems and embodiments for multispectral image demosaicking using deep panchromatic image guided residual interpolation. Embodiments of a ResNet-based deep learning model are disclosed to reconstruct the full-resolution panchromatic image from multispectral filter array (MSFA) mosaic image. In one or more embodiments, the reconstructed deep panchromatic image (DPI) is deployed as the guide to recover the full-resolution multispectral image using a two-pass guided residual interpolation methodology. Experiment results demonstrate that the disclosed method embodiments outperform some state-of-the-art conventional and deep learning demosaicking methods both qualitatively and quantitatively.

Systems and methods for multispectral image demosaicking using deep panchromatic image guided residual interpolation

Provided are systems and methods for imaging with compensation functions. An imaging device comprises a plurality of light source sets, of which two or more light source sets emit light of a same specific spectral range to compensate intensity differences among different spectral ranges. The imaging device can be integrated with a mobile device. A method comprises a subtraction procedure to compensate ambient light effect, a normalization procedure to compensate incidence angle effect, a 3D reconstruction procedure to compensate distance effect, or any combination of these procedures. The method is performed at an imaging device comprising a controller. At least one program is non-transiently stored in the controller and executable by the controller.

Systems and methods for spectral imaging with compensation functions

Systems and methods for imaging using a plurality of light sources is provided. An imaging device comprises a housing, a lens within the housing, a plurality of light source sets attached or integrated into the housing, a two-dimensional pixelated detector within the housing in optical communication with the lens, memory and a controller. Each respective light source set comprises a plurality of lights uniformly radially distributed about the lens. A first light source set in the plurality sets emits light substantially limited to a first spectral range. A second light source set in the plurality of sets emits light substantially limited to a second spectral range other than the first range. At least one program non-transiently stored in the memory and executable by the controller sequentially and independently fired each light source set while collecting light using the pixelated detector. An imaging device can be integrated with a mobile device.

Systems and methods for spectral imaging with a transmitter using a plurality of light sources

A method for capturing and consolidating multi-spectral image data for regions of interest using one or more spectral boost mechanisms or techniques is disclosed. A first multi-spectral or hyperspectral workflow comprising time-stamped two-dimensional pixelated images of a region of interest is obtained by a first computer-enabled imaging device. Each of the images corresponds to image data captured for a respective frequency band of a plurality of frequency bands. First meta data is associated with the images of the first workflow and includes: a plurality of first positions and first orientations of the first imaging device indicating first positional and orientation data for the first imaging device, and indications of a respective frequency band for each of the images of the first workflow. A consolidated multi-spectral workflow for the region of interest is generated, which includes spectrally and spatially consolidating the images of the first workflow using the first meta data.

Growing community of inventors

San Jose, CA, United States of America

Zhihong Pan