Apple Inc.

Frank Doepke

Marius Tico

Todd Stephen Sachs

Marco Zuliani

Ralph T Brunner

Mark Zimmer

David Hayward

Geoff Stahl

Paul M Hubel

Jianping Zhou

Claus Molgaard

Kevin L Hunter

Ricardo Jansson Motta

George E Williams

Hans-Werner Neubrand

Alexander Lindskog

Brett M Keating

Thomas E Bishop

Vincent Y Wong

Ethan J Tira-Thompson

Suk Hwan Lim

Ting Chen

Rolf Toft

Howard Aron Miller

Jason Klivington

Sebastien X Beysserie

Douglas DeVore

Alexis Gatt

Yongfang Liang

Benjamin M Olson

Anita Nariani-Schulze

Ralf Brunner

The invention relates to systems, methods, and computer readable media for responding to a user snapshot request by capturing anticipatory pre-snapshot image data as well as post-snapshot image data. The captured information may be used, depending upon the embodiment, to create archival image information and image presentation information that is both useful and pleasing to a user. The captured information may automatically be trimmed or edited to facilitate creating an enhanced image, such as a moving still image. Varying embodiments of the invention offer techniques for trimming and editing based upon the following: exposure, brightness, focus, white balance, detected motion of the camera, substantive image analysis, detected sound, image metadata, and/or any combination of the foregoing.

Method and apparatus for finding and using video portions that are relevant to adjacent still images

Systems, methods, and computer readable media for adaptively selecting what portion (aka slice) of a first image (aka frame) is selected to overlap and blend with a second frame during frame capture operations are disclosed. In general, for every new frame captured in a sequence the overlap between it and the slice selected from a prior frame may be determined based, at least in part, on sensor output. If the overlap so determined is below a desired threshold, the position of the current frame's slice may be adjusted so as to provide the desired overlap.

Motion adaptive image slice selection

Generating an image with a selected level of background blur includes capturing, by a first image capture device, a plurality of frames of a scene, wherein each of the plurality of frames has a different focus depth, obtaining a depth map of the scene, determining a target object and a background in the scene based on the depth map, determining a goal blur for the background, and selecting, for each pixel in an output image, a corresponding pixel from the focus stack.

Photo-realistic shallow depth-of-field rendering from focal stacks

A graphical user interface (GUI) element permits a user to control an application in both a coarse manner and a fine manner. When a cursor is moved to coincide or overlap the displayed GUI element, parameter adjustment is made at a first (coarse) granularity so that rapid changes to the target parameter can be made (e.g., displayed zoom level, image rotation or playback volume). As the cursor is moved away from the displayed GUI element, parameter adjustment is made at a second (fine) granularity so that fine changes to the target parameter can be made. In one embodiment, the further the cursor is moved from the displayed GUI element, the finer the control.

Granular graphical user interface element

Lens flare mitigation techniques determine which pixels in images of a sequence of images are likely to be pixels affected by lens flare. Once the lens flare areas of the images are determined, unwanted lens flare effects may be mitigated by various approaches, including reducing border artifacts along a seam between successive images, discarding entire images of the sequence that contain lens flare areas, and using tone-mapping to reduce the visibility of lens flare.

Flare detection and mitigation in panoramic images

Special blend operations for wide area-of-view image generation utilizing a 'floating auto exposure' scheme are described. Pixel values in the two images being stitched together are blended within a transition band around a 'seam.' identified in the overlap region between the images after changes in exposure and/or color saturation are accounted for. In some embodiments, changes in exposure and/or color saturation are accounted for through the use of one or more exposure mapping curves, the selection and use of which are based, at least in part, on a determined &#x201c;Exposure Ratio&#x201d; value, i.e., the amount that the camera's exposure settings have deviated from their initial capture settings. In other embodiments, the Exposure Ratio value is also used to determine regions along the seam where either: alpha blending, Poisson blending&#x2014;or a combination of the two&#x2014;should be used to blend in the transitional areas on each side of the seam.

Exposure mapping and dynamic thresholding for blending of multiple images using floating exposure

The techniques disclosed herein may use various sensors to infer a frame of reference for a hand-held device. In fact, with various inertial clues from accelerometer, gyrometer, and other instruments that report their states in real time, it is possible to track a Frenet frame of the device in real time to provide an instantaneous (or continuous) 3D frame-of-reference. In addition to&#x2014;or in place of&#x2014;calculating this instantaneous (or continuous) frame of reference, the position of a user's head may either be inferred or calculated directly by using one or more of a device's optical sensors, e.g., an optical camera, infrared camera, laser, etc. With knowledge of the 3D frame-of-reference for the display and/or knowledge of the position of the user's head, more realistic virtual 3D depictions of the graphical objects on the device's display may be created&#x2014;and interacted with&#x2014;by the user.

Three dimensional user interface effects on a display

This disclosure pertains to devices, methods, and computer readable media for performing positional sensor-assisted panoramic photography techniques in handheld personal electronic devices. Generalized steps that may be used to carry out the panoramic photography techniques described herein include, but are not necessarily limited to: 1.) acquiring image data from the electronic device's image sensor; 2.) performing 'motion filtering' on the acquired image data, e.g., using information returned from positional sensors of the electronic device to inform the processing of the image data; 3.) performing image registration between adjacent captured images; 4.) performing geometric corrections on captured image data, e.g., due to perspective changes and/or camera rotation about a non-center of perspective (COP) camera point; and 5.) 'stitching' the captured images together to create the panoramic scene, e.g., blending the image data in the overlap area between adjacent captured images. The resultant stitched panoramic image may be cropped before final storage.

Positional sensor-assisted perspective correction for panoramic photography

In one embodiment, an image-capturing device includes a camera, one or more motion-estimating devices to detect motion data for the device, and a processing system which is configured to automatically estimate a first direction of a sweep motion of the image-capturing device based on the motion data. The processing system is further configured to automatically estimate a second direction of the sweep motion of the image-capturing device based on images captured by the image-capturing device and to automatically select the first direction or the second direction of the sweep motion.

Methods and systems for determining a direction of a sweep motion

Techniques to detect subject and camera motion in a set of consecutively captured image frames are disclosed. More particularly, techniques disclosed herein temporally track two sets of downscaled images to detect motion. One set may contain higher resolution and the other set lower resolution of the same images. For each set, a coefficient of variation may be computed across the set of images for each sample in the downscaled image to detect motion and generate a change mask. The information in the change mask can be used for various applications, including determining how to capture a next image in the sequence.

Scene stability detection

In personal electronic devices including digital imaging capability, methods, devices and computer readable media are described for determining when image capture operations may benefit from using high dynamic range imaging (HDRI) operations. In general, techniques are disclosed for analyzing an image's luminosity and/or color/tonal histograms to automatically determine when HDRI operations can benefit scene capture. If the determination that HDRI operations can improve scene capture, the user is so notified.

High dynamic range transition

Generating a focus stack, including receiving initial focus data that identifies a plurality of target depths, positioning a lens at a first position to capture a first image at a first target depth of the plurality of target depths, determining, in response to capturing the first image and prior to capturing additional images, a sharpness metric for the first image, capturing, in response to determining that the sharpness metric for the first image is an unacceptable value, a second image at a second position based on the sharpness metric, wherein the second position is not included in the plurality of target depths, determining that a sharpness metric for the second image is an acceptable value, and generating a focus stack using the second image.

Optimizing capture of focus stacks

Traditionally, time-lapse videos are constructed from images captured at time intervals called 'temporal points of interests' or 'temporal POIs.' Disclosed herein are systems and methods of constructing improved, motion-stabilized time-lapse videos using temporal points of interest and image similarity comparisons. According to some embodiments, a &#x201c;burst&#x201d; of images may be captured, centered around the aforementioned temporal points of interest. Then, each burst sequence of images may be analyzed, e.g., by performing an image similarity comparison between each image in the burst sequence and the image selected at the previous temporal point of interest. Selecting the image from a given burst that is most similar to the previous selected image, while minimizing the amount of motion with the previous selected image, allows the system to improve the quality of the resultant time-lapse video by discarding &#x201c;outlier&#x201d; or other undesirable images captured in the burst sequence and motion stabilizing the selected image.

Growing community of inventors

San Jose, CA, United States of America

Frank Doepke