Nikon Corporation

Stmicroelectronics Gmbh

Niko Corporation

Li Hong

Mark K Takita

W Thomas Novak

Radka Tezaur

Gazi Ali

George Q Chen

Peter J McGuinness

Simant Dube

Maki Suzuki

Itaru Homma

Hany Farid

Tetsuji Kamata

George-Qian Chen

Deblurring a blurry image () includes the steps of (i) computing a spatial mask (); (ii) computing a modified blurry image () using the blurry image () and the spatial mask (); and (iii) computing a latent sharp image () using the modified blurry image () and a point spread function (). Additionally, the image () can be analyzed to identify areas of the image () that are suitable for point spread function estimation. Moreover, a region point spread function () can be analyzed to classify the point spread function(s) as representing (i) motion blur, (ii) defocus blur, or (iii) mixed motion blur and defocus blur. A point spread function () can also be estimated.

Algorithm and device for image processing

A method for detecting a main subject in an image comprises the steps of: (i) computing a plurality of saliency features from the image () with a control system (); (ii) generating a spatial weight map for each of a plurality of image segments of the image () with the control system (); (iii) adjusting the plurality of saliency features via the spatial weight map to generate a plurality of adjusted saliency features; (iv) combining at least two of the plurality of adjusted saliency features to generate a saliency map of the image (); and (v) extracting the main subject from the saliency map of the image (). Additionally, the plurality of saliency features can include at least two of a sharp/blur saliency feature, a spectral residual saliency feature, a color spatial distribution saliency feature, and one or more color contrast saliency features.

Method of detecting a main subject in an image

A group point spread function () for a blurry image () can be determined by dividing the blurry image () into a plurality of image regions (), estimating a region point spread function () for at least two of the image regions (); and utilizing at least two of the region point spread functions () to determine the group point spread function (). The group point spread function () can be determined by the decomposition of the estimated region point spread functions () into some basis function, and subsequently determining a representative coefficient from the basis functions of the region point spread functions () to generate the group point spread function ().

Globally dominant point spread function estimation

A method for classifying a test image () includes the steps of building a classifier (), and classifying the test image () with the classifier (). After the test image () is classified, the test image () can be subsequently processed (e.g. deblurred) with improved accuracy. The classifier () can classify and distinguish between PSF features associated with motion blurred images, and PSF features associate with defocus blurred images. The classifier () can be built using a plurality of training images (), and extracting one or more training features from each the training images (). The PSF features can include image moments of the point spread function, and/or geometric features of the point spread function.

Point spread function classification using structural properties

A system () for detecting whether an image () is blurred performs the steps of: identifying valid light trails in the test image (), and classifying the test image as blurred if there is a sufficient number of matching valid light trails oriented in a repetitive pattern (). The system () is designed to automatically detect whether a night scene image () is blurred based on the existence and characteristics of any light trails () in the captured image (). Because the method extracts the characteristics of the light trails () and analyzes these light trails (), it accurately detects blur in the challenging night scene images and has a relatively low false positive rate.

Blur detection system for night scene images

A method for processing a blurred image () that includes a captured light trail () includes the steps of: (i) displaying the blurred image () on an image display (); (ii) manually identifying an image patch () that includes the captured light trail () in the displayed blurred image (); (iii) calculating a point spread function for the captured light trail () with a control system (); and (iv) deblurring at least a portion of the blurred image () with the control system () utilizing the calculated point spread function.

Motion blur estimation and restoration using light trails

A method for predicting whether a test image () is sharp or blurred includes the steps of: training a sharpness classifier () to discriminate between sharp and blurred images, the sharpness classifier () being trained based on a set of training sharpness features () computed from a plurality of training images (), the set of training sharpness features () for each training image () being computed by (i) resizing each training image () by a first resizing factor; (ii) identifying texture regions () in the resized training image; and (iii) computing the set of sharpness features in the training image () from the identified texture regions; and applying the trained sharpness classifier () to the test image () to determine if the test image () is sharp or blurred based on a set of test sharpness features () computed from the test image (), the set of test sharpness features () for each test image () being computed by (i) resizing the test image () by a second resizing factor that is different than the first resizing factor; (ii) identifying texture regions () in the resized test image; and (iii) computing the set of sharpness features in the test image () from the identified texture regions.

Image sharpness classification system

An image apparatus for providing an estimated illuminant of a scene and subsequently providing an adjusted image of the scene includes a capturing system and an illuminant database. The capturing system captures information for an input image of the scene. The input image is defined by a plurality of input colors. The illuminant database includes a separate subspace of information for a plurality of possible illuminants. The subspaces can be created from multi-dimensional color information from its respective possible illuminant that is computed using Principal Components Analysis. This allows for more accurate illuminant estimation of the scene and the use of Principal Components Analysis to generate the one or more subspaces allows for accurate and relatively easy generation of the subspaces. Additionally, the image apparatus can include a control system that compares the input colors to the subspaces in the illuminant database.

Image apparatus for principal components analysis based illuminant estimation

A system for providing an adjusted image of a scene includes an optical assembly, a capturing system coupled to the optical assembly, and a control system. The optical assembly is adjustable to alternatively be focused on a first focal area and a second focal area that is different than the first focal area. The capturing system captures a first frame of the scene when the optical assembly is focused at the first focal area, and a second frame of the scene when the optical assembly is focused at the second focal area. The first frame includes a plurality of first pixels and the second frame includes a plurality of second pixels. The control system analyzes the first frame and the second frame and utilizes graph cuts techniques to assign a depth label to at least a portion of the first frame.

Image segmentation from focus varied images using graph cuts

An image apparatus () for providing an estimated final illuminant of a scene () and subsequently providing an adjusted image () of a scene () includes a capturing system () and a control system (). The capturing system () captures information for an input image () of the scene () while at an apparatus setting. The input image () is defined by a plurality of input colors. The control system () can compare at least one of the input colors to an illuminant database () that includes an illuminant gamut of observable colors for a plurality of possible illuminants, and can evaluate the apparatus setting to select one of the possible illuminants to be the estimated final illuminant. Examples of possible apparatus settings include an aperture size and an exposure time. With this design, the present invention is better able to distinguish between some possible outdoor illuminants and some possible indoor illuminants with improved accuracy. Additionally, the control system () can evaluate a consistency matching error for the plurality of possible illuminants to determine if the scene () has intrinsic color casting.

Automatic illuminant estimation that incorporates apparatus setting and intrinsic color casting information

A method for predicting whether a test image () is sharp or blurred includes the steps of: providing a sharpness classifier () that is trained to discriminate between sharp and blurred images; computing a set of sharpness features () for the test image () by (i) generating a high pass image () from the test image (), (ii) generating a band pass image () from the test image (), (iii) identifying textured regions () in the high pass image, (iv) identifying texture regions () in the band pass image, and (v) evaluating the identified textured regions in the high pass image and the band pass image to compute the set of test sharpness features (); and evaluating the sharpness features using the sharpness classifier () to estimate if the test image () is sharp or blurry ().

Image sharpness classification  system

A system () for providing an adjusted image () of a scene () from a noisy captured image () includes a control system () that provides the adjusted image (). The control system () can create a de-noised image () from the captured image (), determine a details layer () from the captured image (), and combine information from the details layer () with the de-noised image () to provide the adjusted image ().

Method and device for image noise compensation with image enhancement

A system () for providing an adjusted image () of a scene () from a blurred captured image () includes a control system () that provides the adjusted image (). The control system () can create a downsampled blurred image () from the captured image (), deblur the downsampled blurred image () to create a downsampled deblurred image (), and upsample the downsampled deblurred image () to create the adjusted image (). With this design, because the deblurring is performed on the downsampled blurred image (), the computational and memory costs are substantially less than if this task is performed on the original captured image (). Further, the control system () can create the adjusted image () using a joint bilateral filter that combines information from the blurred captured image () and the downsampled deblurred image ().

Method and device for image deblurring using joint bilateral filtering

The present invention is directed to a method for detecting or predicting () whether a test image is blurred. In one embodiment, the method includes extracting a training statistical signature () that is based on a plurality of data features () from a training image set (), the training image set () including a sharp image () and a blurry image (); training a classifier () to discriminate between the sharp image () and the blurry image () based on the training statistical signature; and applying () the trained classifier to a test image that is not included in the training image set () to predict whether the test image is sharp () or blurry (). The step of extracting can include measuring one or more statistical moments () for various levels (L-L), estimating a covariance () between adjacent levels (L-L), and/or extracting various metadata features () from the images (). The step of training () can include training a non-linear support vector machine () or a linear discriminant analysis () on the training statistical signature of the training image set ().

Device and method for detecting whether an image is blurred

A method for estimating whether an image () is either blurred or sharp includes the steps of (i) determining an image gradient histogram distrbution () of at least a portion of the image (), (ii) comparing at least a portion of the image gradient histogram distribution () to a Gaussian model (), and (iii) estimating a defocus blur size of at least a portion of the image () Additional methods for evaluating whether the image () has motion blur are provided herein For example, a dominant gradient orientation can be detected by histogram analysis (), and the dominant gradient orientation () can be used to determine a possible motion blur direction () Alternatively, a dominant gradient orientation can be detected using principal component analysis ('PCA') on the image.

Growing community of inventors

San Diego, CA, United States of America

Li Hong