Apple Inc.

Sebastien X Beysserie

Jianping Zhou

George E Williams

Rolf Toft

Damien J Thivent

Rudolph Van Der Merwe

Jason Klivington

Stephane S Ben Soussan

Richard Lawrence Baer

Xiaoxing Li

Anders M Holtsberg

Martin Lindberg

Jozef Kruger

Florian Ciurea

Todd Stephen Sachs

Frank Doepke

Marco Zuliani

Claus Molgaard

Tao Zhang

Feng Li

Ahmed Tashrif Kamal

Ethan J Tira-Thompson

Benjamin M Olson

Ali-Amir Aldan

Saeed Izadi

Devices, methods, and non-transitory computer readable media are disclosed herein to repair or mitigate the appearance of unwanted reflection artifacts in captured video image streams. These unwanted reflection artifacts often present themselves as brightly-colored spots, circles, rings, or halos that reflect the shape of a bright light source in the captured image. These artifacts, also referred to herein as 'ghosts' or “green ghosts” (due to often having a greenish tint), are typically located in regions of the captured images where there is not actually a bright light source located in the image. In fact, such unwanted reflection artifacts often present themselves on the image sensor across the principal point of the lens from where the actual bright light source in the captured image is located. Such devices, methods and computer readable media may be configured to detect, track, and repair such unwanted reflection artifacts in an intelligent and efficient fashion.

Efficient unwanted reflection artifact mitigation in videos for embedded devices

Devices, methods, and non-transitory program storage devices are disclosed herein to provide for improved perspective distortion correction for wide field of view (FOV) video image streams. The techniques disclosed herein may be configured, such that the distortion correction applied to requested region of interest (ROI) portions taken from individual images of the wide FOV video image stream smoothly transitions between applying different distortion correction to ROIs, depending on their respective FOVs. In particular, the techniques disclosed herein may modify the types and/or amounts of perspective distortion correction applied, based on the FOVs of the ROIs, as well as their location within the original wide FOV video image stream. In some cases, additional perspective distortion correction may also be applied to account for tilt in an image capture device as the wide FOV video image stream is being captured and/or the unwanted inclusion of 'invalid' pixels from the wide FOV image.

Fast distortion correction for wide field of view (FOV) cameras

Techniques are described for automatically selecting between multiple image capture subsystems with overlapping fields of view but different optical properties. A selection may be made by estimating a plurality of operational characteristics of an image capture event, and, based on those estimates, selecting a primary image capture subsystem for the image capture event. For example, in a device such as a cellphone comprising two capture subsystems, each subsystem including a lens system and sensor system where each subsystem has a different fixed optical zoom parameter, a subsystem can be chosen based on a combination of desired zoom value, estimated focus distance, and estimated scene brightness.

Switchover control techniques for dual-sensor camera system

Various techniques are disclosed herein, which comprise obtaining a first stream of images from a first camera and a second stream of images from a second camera, wherein the first camera and second camera have different poses, and at least a portion of the image streams are captured concurrently. Next, a first stabilized trajectory and corresponding set of corrections used to stabilize the first stream of images are calculated. Based on determined differences between the poses of the first camera and second camera during the capture of the first and second streams of images, a second set of corrections to be applied to the second stream of images are derived to match the first stabilized trajectory. If desired, e.g., in response to determining that the stabilized first stream of images does not meet a stabilization criterion, the second stream of images may be stabilized using the second set of corrections.

Multi-camera video stabilization techniques

Techniques to improve a digital image capture device's ability to stabilize a video stream&#x2014;while enforcing desired stabilization constraints on particular images in the video stream&#x2014;are presented that utilize an overscan region and a look-ahead technique enabled by buffering a number of video input frames before generating a first stabilized video output frame. More particularly, techniques are disclosed for buffering an initial number of input frames so that a 'current' frame can use motion data from both 'past' and &#x201c;future&#x201d; frames to adjust the value of a stabilization strength parameter and/or the weighted contribution of particular frames from the buffer in the determination of stabilization motion values for the current frame. Such techniques keep the current frame within its overscan and ensure that the stabilization constraints are enforced, while maintaining desired smoothness in the video stream. In some embodiments, the stabilization constraint may comprise a maximum allowed frame displacement.

Video image stabilization with enforced stabilization constraints

Techniques to improve a digital image capture device's ability to stabilize a video stream are presented. According to some embodiments, improved stabilization of captured video frames is provided by intelligently harnessing the complementary effects of both optical image stabilization (OIS) and electronic image stabilization (EIS). In particular, OIS may be used to remove intra-frame motion blur that is typically lower in amplitude and dominates with longer integration times, while EIS may be used to remove residual unwanted frame-to-frame motion that is typically larger in amplitude. The techniques disclosed herein may also leverage information provided from the image capture device's OIS system to perform improved motion blur-aware video stabilization strength modulation, which permits better video stabilization performance in low light conditions, where integration times tend to be longer, thus leading to a greater amount of motion blurring in the output stabilized video.

Combined optical and electronic image stabilization

Techniques to permit a digital image capture device to stabilize a video stream in real-time (during video capture operations) are presented. In general, techniques are disclosed for stabilizing video images using an overscan region and a look-ahead technique enabled by buffering a number of video input frames before generating a first stabilized video output frame. (Capturing a larger image than is displayed creates a buffer of pixels around the edge of an image; overscan is the term given to this buffer of pixels.) More particularly, techniques are disclosed for buffering an initial number of input frames so that a 'current' frame can use motion data from both 'past' and &#x201c;future&#x201d; frames to adjust the strength of a stabilization metric value so as to keep the current frame within its overscan. This look-ahead and look-behind capability permits a smoother stabilizing regime with fewer abrupt adjustments.

Video image stabilization

Techniques to improve a digital image capture device's ability to stabilize a video stream in low light capture conditions are presented. In general, techniques are disclosed for stabilizing video images using an overscan region and a look-ahead technique enabled by buffering a number of video input frames before generating a first stabilized video output frame. More particularly, techniques are disclosed for buffering an initial number of input frames so that a 'current' frame can use time stamped image capture device motion data from both 'past' and &#x201c;future&#x201d; frames to adjust the strength of a stabilization metric value so as to keep the current frame within its overscan. The calculated adjustments to the strength of the stabilization metric value for a particular frame may be limited to a target max strength value that is determined based, at least in part, on an estimated amount of motion blur in the current frame.

Low light video image stabilization strength modulation

A method to improve the efficiency of the detection and tracking of machine-readable objects is disclosed. The properties of image frames may be pre-evaluated to determine whether a machine-readable object, even if present in the image frames, would be likely to be detected. After it is determined that one or more image frames have properties that may enable the detection of a machine-readable object, image data may be evaluated to detect the machine-readable object. When a machine-readable object is detected, the location of the machine-readable object in a subsequent frame may be determined based on a translation metric between the image frame in which the object was identified and the subsequent frame rather than a detection of the object in the subsequent frame. The translation metric may be identified based on an evaluation of image data and/or motion sensor data associated with the image frames.

Efficient machine-readable object detection and tracking

Traditionally, time-lapse videos are constructed from images captured at time intervals called 'temporal points of interests' or 'temporal POIs.' Disclosed herein are systems and methods of constructing improved, motion-stabilized time-lapse videos using temporal points of interest and image similarity comparisons. According to some embodiments, a &#x201c;burst&#x201d; of images may be captured, centered around the aforementioned temporal points of interest. Then, each burst sequence of images may be analyzed, e.g., by performing an image similarity comparison between each image in the burst sequence and the image selected at the previous temporal point of interest. Selecting the image from a given burst that is most similar to the previous selected image, while minimizing the amount of motion with the previous selected image, allows the system to improve the quality of the resultant time-lapse video by discarding &#x201c;outlier&#x201d; or other undesirable images captured in the burst sequence and motion stabilizing the selected image.

Time-lapse video capture with optimal image stabilization

An apparatus, method, and computer-readable medium for motion sensor-based video stabilization. A motion sensor may capture motion data of a video sequence. A controller may compute instantaneous motion of the camera for a current frame of the video sequence. The controller may compare the instantaneous motion to a threshold value representing a still condition and reduce a video stabilization strength parameter for the current frame if the instantaneous motion is less than the threshold value. A video stabilization unit may perform video stabilization on the current frame according to the frame's strength parameter.

Growing community of inventors

San Jose, CA, United States of America

Sebastien X Beysserie