Apple Inc.

Intel Corporation

Texas Instruments Corporation

Jianping Zhou

Sebastien X Beysserie

Rolf Toft

George E Williams

Damien J Thivent

Xiaoxing Li

Paul M Hubel

Marius Tico

Marco Zuliani

Rudolph Van Der Merwe

Todd Stephen Sachs

Frank Doepke

Anita Nariani Schulze

Stephane S Ben Soussan

Ravishankar Iyer

Richard Lawrence Baer

Florian Ciurea

Rameshkumar G Illikkal

Donald K Newell

Claus Molgaard

Tao Zhang

Jason Klivington

Geoffrey T Anneheim

Ahmed Tashrif Kamal

Anders M Holtsberg

Feng Li

Martin Lindberg

Yong Zhang

Benjamin M Olson

Jozef Kruger

Ralph T Brunner

Oscar Nestares

Christopher L Mills

Wei Sun

Madhu S Athreya

Brandon J Corey

Santiago Alban

Abhishek Dhanda

Richard David Seely

Kevin L Hunter

Kevin A Gross

Walker J Eagleston

Ethan J Tira-Thompson

Sheethal Bhat

Ali-Amir Aldan

Jia Xue

Libin Sun

Saeed Izadi

George William

Devices, methods, and non-transitory computer readable media are disclosed herein to repair or mitigate the appearance of unwanted reflection artifacts in captured video image streams. These unwanted reflection artifacts often present themselves as brightly-colored spots, circles, rings, or halos that reflect the shape of a bright light source in the captured image. These artifacts, also referred to herein as 'ghosts' or “green ghosts” (due to often having a greenish tint), are typically located in regions of the captured images where there is not actually a bright light source located in the image. In fact, such unwanted reflection artifacts often present themselves on the image sensor across the principal point of the lens from where the actual bright light source in the captured image is located. Such devices, methods and computer readable media may be configured to detect, track, and repair such unwanted reflection artifacts in an intelligent and efficient fashion.

Efficient unwanted reflection artifact mitigation in videos for embedded devices

Various embodiments include implementing optical image stabilization (OIS) in a camera module. For example, a controller may control an actuator of the camera module to move a lens group and/or an image sensor of the camera module to provide OIS movement. In various embodiments, OIS movement control is implemented according to a region-based blur reduction model that effects a greater reduction in blur associated with a target region of an image sensor, relative to one or more other regions of the image sensor.

Optical image stabilization with region-based blur reduction

Devices, methods, and non-transitory program storage devices are disclosed herein to provide for improved perspective distortion correction for wide field of view (FOV) video image streams. The techniques disclosed herein may be configured, such that the distortion correction applied to requested region of interest (ROI) portions taken from individual images of the wide FOV video image stream smoothly transitions between applying different distortion correction to ROIs, depending on their respective FOVs. In particular, the techniques disclosed herein may modify the types and/or amounts of perspective distortion correction applied, based on the FOVs of the ROIs, as well as their location within the original wide FOV video image stream. In some cases, additional perspective distortion correction may also be applied to account for tilt in an image capture device as the wide FOV video image stream is being captured and/or the unwanted inclusion of 'invalid' pixels from the wide FOV image.

Fast distortion correction for wide field of view (FOV) cameras

Devices, methods, and non-transitory program storage devices are disclosed to provide enhanced images in multi-camera systems, e.g., by using information from images captured by cameras with different properties in terms of optics and/or sensors. In one embodiment, the techniques comprise: obtaining a first image from a first image capture device, wherein the first image has a first field of view (FOV) and a first set of quality characteristics; obtaining a second image from a second image capture device, wherein the second image has a second FOV and a second set of quality characteristics, and wherein the second FOV partially overlaps the first FOV; obtaining a neural network that produces a modified second image having a modified second set of quality characteristics determined by the neural network attempting to match the first set of quality characteristics; and generating an output image based, at least in part, on the modified second image.

Systems and methods for multi-sensor image enhancement

Systems and methods for stitching videos are disclosed. Image-based registration between frames from a first video source and frames from a second video source is performed at a first rate. Calibration-based registration between frames from the first video source and frames from the second video source are performed at a second rate higher than the first rate. Then, for a first frame from the first video source for which calibration-based registration data and image-based registration data have been generated, a stitching transform that maps the first frame to a counterpart frame from the second video source based on image-based registration data is generated. A delta transform from the image-based registration data and the calibration-based registration data at the first frame is also derived. For a subsequent frame from the first video source for which calibration-based registration data have been generated, but no image-based registration data have been generated, a stitching transform that maps the subsequent frame to a counterpart frame from the second video source based on the calibration-based registration data and the delta transform is generated. Frames from the first video source and frames from the second video source are stitched according to their respective generated stitching transforms.

Seamless preview stitching for multiple cameras

Techniques are described for automatically selecting between multiple image capture subsystems with overlapping fields of view but different optical properties. A selection may be made by estimating a plurality of operational characteristics of an image capture event, and, based on those estimates, selecting a primary image capture subsystem for the image capture event. For example, in a device such as a cellphone comprising two capture subsystems, each subsystem including a lens system and sensor system where each subsystem has a different fixed optical zoom parameter, a subsystem can be chosen based on a combination of desired zoom value, estimated focus distance, and estimated scene brightness.

Switchover control techniques for dual-sensor camera system

Electronic devices, computer readable storage media, and related methods are disclosed herein that are configured to stitch together images captured by multiple image capture devices of an image capture system. In particular, various techniques are employed to intelligently extend (and, optionally, smooth) the correspondence mapping between first and second images captured by image capture devices having different fields of view, e.g., fields of view that are at least partially overlapping and at least partially non-overlapping. The techniques may also include determining a 'transitional' correspondence in a transitional region between the overlapping and non-overlapping regions of the fields of view, as well as performing one or more appearance correction operations to account for the different properties of the different image capture devices used to capture the first and second images. The techniques described herein may be employed to produce enhanced output images in either the still image or the video context.

Field of view extension in a stereo camera system

Various techniques are disclosed herein, which comprise obtaining a first stream of images from a first camera and a second stream of images from a second camera, wherein the first camera and second camera have different poses, and at least a portion of the image streams are captured concurrently. Next, a first stabilized trajectory and corresponding set of corrections used to stabilize the first stream of images are calculated. Based on determined differences between the poses of the first camera and second camera during the capture of the first and second streams of images, a second set of corrections to be applied to the second stream of images are derived to match the first stabilized trajectory. If desired, e.g., in response to determining that the stabilized first stream of images does not meet a stabilization criterion, the second stream of images may be stabilized using the second set of corrections.

Multi-camera video stabilization techniques

Techniques to improve a digital image capture device's ability to stabilize a video stream&#x2014;while enforcing desired stabilization constraints on particular images in the video stream&#x2014;are presented that utilize an overscan region and a look-ahead technique enabled by buffering a number of video input frames before generating a first stabilized video output frame. More particularly, techniques are disclosed for buffering an initial number of input frames so that a 'current' frame can use motion data from both 'past' and &#x201c;future&#x201d; frames to adjust the value of a stabilization strength parameter and/or the weighted contribution of particular frames from the buffer in the determination of stabilization motion values for the current frame. Such techniques keep the current frame within its overscan and ensure that the stabilization constraints are enforced, while maintaining desired smoothness in the video stream. In some embodiments, the stabilization constraint may comprise a maximum allowed frame displacement.

Video image stabilization with enforced stabilization constraints

Techniques to improve a digital image capture device's ability to stabilize a video stream are presented. According to some embodiments, improved stabilization of captured video frames is provided by intelligently harnessing the complementary effects of both optical image stabilization (OIS) and electronic image stabilization (EIS). In particular, OIS may be used to remove intra-frame motion blur that is typically lower in amplitude and dominates with longer integration times, while EIS may be used to remove residual unwanted frame-to-frame motion that is typically larger in amplitude. The techniques disclosed herein may also leverage information provided from the image capture device's OIS system to perform improved motion blur-aware video stabilization strength modulation, which permits better video stabilization performance in low light conditions, where integration times tend to be longer, thus leading to a greater amount of motion blurring in the output stabilized video.

Combined optical and electronic image stabilization

Systems, methods, and computer readable media to rapidly identify and track an arbitrary sized object through a temporal sequence of frames is described. The object being tracked may initially be identified via a specified or otherwise known region-of-interest (ROI). A portion of that ROI can be used to generate an initial or reference histogram and luminosity measure, metrics that may be used to identify the ROI in a subsequent frame. For a frame subsequent to the initial or reference frame, a series of putative ROIs (each having its own location and size) may be identified and the 'best' of the identified ROIs selected. As used here, the term 'best' simply means that the more similar two frames' histograms and luminosity measures are, the better one is with respect to the other.

Fast histogram-based object tracking

Techniques to identify and track a pre-identified region-of-interest (ROI) through a temporal sequence of frames/images are described. In general, a down-sampled color gradient (edge map) of an arbitrary sized ROI from a prior frame may be used to generate a small template. This initial template may be used to identify a region of a new or current frame that may be overscan and used to create a current frame's edge map. By comparing the prior frame's template to the current frame's edge map, a cost value or image may be found and used to identify the current frame's ROI center. The size of the current frame's ROI may be found by varying the size of putative new ROIs and testing for their congruence with the prior frame's template. Subsequent ROI's for subsequent frames may be identified to, effectively, track an arbitrarily sized ROI through a sequence of video frames.

Fast template-based tracking

This disclosure pertains to devices, methods, and computer readable media for performing positional sensor-assisted panoramic photography techniques in handheld personal electronic devices. Generalized steps that may be used to carry out the panoramic photography techniques described herein include, but are not necessarily limited to: 1.) acquiring image data from the electronic device's image sensor; 2.) performing 'motion filtering' on the acquired image data, e.g., using information returned from positional sensors of the electronic device to inform the processing of the image data; 3.) performing image registration between adjacent captured images; 4.) performing geometric corrections on captured image data, e.g., due to perspective changes and/or camera rotation about a non-center of perspective (COP) camera point; and 5.) 'stitching' the captured images together to create the panoramic scene, e.g., blending the image data in the overlap area between adjacent captured images. The resultant stitched panoramic image may be cropped before final storage.

Positional sensor-assisted perspective correction for panoramic photography

Growing community of inventors

Fremont, CA, United States of America

Jianping Zhou