Apple Inc.

Hewlett-packard Development Company, L.p.

Google Inc.

Leland Stanford Junior University

Suk Hwan Lim

D Amnon Silverstein

Sheng Lin

Guy Cote

David R Pope

Joseph P Bratt

Timothy John Millet

Ting Chen

Pavel Kisilev

Bin Ni

Shun Wai Go

Feng Tang

Mark Zimmer

Haitao Guo

Christopher L Mills

Blaise Hilary Aguera-Arcas

Muge Wang

Junji Sugisawa

Steven David Hordley

Jian-Ming Jin

Daniel R Tretter

Jian Fan

Jerry J Liu

Jonathan Yen

Graham D Finlayson

Nelson Liang An Chang

Weichun Ku

Samson J Liu

Ramin Samadani

Qian Lin

Chung Chun Wan

Szepo Robert Hung

Choon Ping Chng

David Hayward

Ron Maurer

Yuhong Xiong

Nina Bhatti

Jaewon Shin

Eamonn O'Brien-Strain

Simon Wolfenden Butler

Joseph Anthony Petolino, Jr

Touraj Tajbakhsh

Parag Mulendra Joshi

Hui-Man Hou

Sheng-Wen Yang

Li-Wei Zheng

Debargha Mukherjee

Xinqiao Liu

Ralph T Brunner

George H Forman

Hui Chao

Peter G Hwang

Niranjan Damera-Venkata

David Chen

Andrew Kenneth John McMahon

Huitao Luo

Frank Doepke

Peng Wu

Ping Luo

Guy Côté

Frederic Cao

Mei Chen

Abbas El Gamal

Munehiro Mori

Yuli Gao

Hsin-I Cindy Liu

Huizhong Chen

Li-Mei Jiao

Benjamin M Olson

Adam C Petruszka

Anita Nariani-Schulze

Terence C Ngai

The present disclosure generally relates to systems and methods for image data processing. In certain embodiments, an image processing pipeline may be configured to receive a frame of the image data having a plurality of pixels acquired using a digital image sensor. The image processing pipeline may then be configured to determine a first plurality of correction factors that may correct each pixel in the plurality of pixels for fixed pattern noise. The first plurality of correction factors may be determined based at least in part on fixed pattern noise statistics that correspond to the frame of the image data. After determining the first plurality of correction factors, the image processing pipeline may be configured to apply the first plurality of correction factors to the plurality of pixels, thereby reducing the fixed pattern noise present in the plurality of pixels.

Systems and methods for reducing fixed pattern noise in image data

The present disclosure generally relates to systems and methods for image data processing. In certain embodiments, an image processing pipeline may be configured to receive a frame of the image data having a plurality of pixels acquired using a digital image sensor. The image processing pipeline may then be configured to determine a first plurality of correction factors that may correct each pixel in the plurality of pixels for fixed pattern noise. The first plurality of correction factors may be determined based at least in part on fixed pattern noise statistics that correspond to the frame of the image data. After determining the first plurality of correction factors, the image processing pipeline may be configured to configured to apply the first plurality of correction factors to the plurality of pixels, thereby reducing the fixed pattern noise present in the plurality of pixels.

Systems and method for reducing fixed pattern noise in image data

The present disclosure provides image capture devices and associated methods that feature intelligent use of hardware-generated statistics. An example image capture device can include an imaging hardware pipeline that generates frames of imagery. The imaging hardware pipeline can generate one or more hardware-generated statistics based at least in part on, for example, the raw image data captured by the image sensor or intermediate image data within the pipeline. The image capture device can analyze the hardware-generated statistics to determine one or more metrics for the raw image data or the image. The image capture device can determine a downstream operation of the image capture device relative to the image based at least in part on the metrics determined from the hardware generated statistics.

Image capture devices featuring intelligent use of lightweight hardware-generated statistics

In an embodiment, an electronic device may be configured to capture still frames during video capture but may capture the still frames in the 4&#xd7;3 aspect ratio and at higher resolution than the 16&#xd7;9 aspect ratio video frames. The device may interleave high resolution, 4&#xd7;3 frames and lower resolution 16&#xd7;9 frames in the video sequence, and may capture the nearest higher resolution, 4&#xd7;3 frame when the user indicates the capture of a still frame. Alternatively, the device may display 16&#xd7;9 frames in the video sequence, and then expand to 4&#xd7;3 frames when a shutter button is pressed. The device may capture the still frame and return to the 16&#xd7;9 video frames responsive to a release of the shutter button.

Modeless video and still frame capture

An image signal processor may include a pixel defect correction component that tracks defect history for frames captured by an image sensor and applies the history when identifying and correcting defective pixels in a frame. The component maintains a defect pixel location table that includes a defect confidence value for pixels of the image sensor. The component identifies defective pixels in a frame, for example by comparing each pixel's value to the values of its neighbor pixels. If a pixel is detected as defective, its defect confidence value may be incremented. Otherwise, the value may be decremented. If a pixel's defect confidence value is over a defect confidence threshold, the pixel is considered defective and thus may be corrected. If a pixel's defect confidence value is under the threshold, the pixel is considered not defective and thus may not be corrected even if the pixel was detected as defective.

Correcting pixel defects based on defect history in an image processing pipeline

Embodiments of the present disclosure relate to a configurable convolution engine that receives configuration information to perform convolution or its variant operations on streaming input data of various formats. To process streaming input data, input data of multiple channels are received and stored in an input buffer circuit in an interleaved manner. Data values of the interleaved input data are retrieved and forwarded to multiplier circuits where multiplication with a corresponding filter element of a kernel is performed. Varying number of kernels with different sizes and sparsity can also be used for the convolution operations.

Configurable convolution engine

An image processing pipeline may dynamically determine filtering strengths for noise filtering of image data. Statistics may be collected for an image at an image processing pipeline. The statistics may be accessed and evaluated to generate a filter strength model that maps respective filtering strengths to different portions of the image. A noise filter may determine a filtering strength for image data received at the noise filter according to the filter strength model. The noise filter may then apply a filtering technique according to the determined filtering strength.

Dynamically determining filtering strength for noise filtering in image processing

Embodiments relate to a histogram-of-oriented gradients (HOG) module. The HOG module is implemented in hardware rather than software. The HOG module applies an algorithm to an image to identify gradient orientation in localized portions of the image. The HOG module creates a histogram-of orientation gradients based on the identified gradient orientations.

Configurable histogram-of-oriented gradients (HOG) processor

An apparatus is described. The apparatus includes a smart image sensor having a memory and a processor that are locally integrated with an image sensor. The memory is to store first program code to be executed by the processor. The memory is coupled to the image sensor and the processor. The memory is to store second program code to be executed by the processor. The first program code is to cause the smart image sensor to perform an analysis on one or more images captured by the image sensor. The analysis identifies a region of interest within the one or more images with machine learning from previously captured images. The second program code is to cause the smart image sensor to change an image sensing and/or optical parameter in response to the analysis of the one or more images performed by the execution of the first program code.

Smart image sensor having integrated memory and processor

An image processing pipeline may perform temporal filtering on independent color channels in image data. A filter weight may be determined for a given pixel received at a temporal filter. The filter weight may be determined for blending a value of a channel in a full color encoding of the given pixel with a value of the same channel for a corresponding pixel in a previously filtered reference image frame. In some embodiments, the filtering strength for the channel may be determined independent from the filtering strength of another channel in the full color encoding of the given pixel. Spatial filtering may be applied to a filtered version of the given pixel prior to storing the given pixel as part of a new reference image frame.

Temporal filtering of independent color channels in image data

Growing community of inventors

Mountain View, CA, United States of America

Suk Hwan Lim