Apple Inc.

Alexander Lindskog

Michael Wish Tao

Frank Doepke

Thomas E Bishop

Mark N Jouppi

Ralph T Brunner

Claus Molgaard

Alexandre Naaman

Richard David Seely

Geoffrey T Anneheim

Ralf Brunner

This disclosure relates to techniques for generating robust depth estimations for captured images using semantic segmentation. Semantic segmentation may be defined as a process of creating a mask over an image, wherein pixels are segmented into a predefined set of semantic classes. Such segmentations may be binary (e.g., a 'person pixel' or a 'non-person pixel') or multi-class (e.g., a pixel may be labelled as: &#x2018;person,&#x2019; &#x2018;dog,&#x2019; &#x2018;cat,&#x2019; etc.). As semantic segmentation techniques grow in accuracy and adoption, it is becoming increasingly important to develop methods of utilizing such segmentations and developing flexible techniques for integrating segmentation information into existing computer vision applications, such as depth and/or disparity estimation, to yield improved results in a wide range of image capture scenarios. In some embodiments, an optimization framework may be employed to optimize a camera device's initial scene depth/disparity estimates that employs both semantic segmentation and color regularization in a robust fashion.

Robust use of semantic segmentation for depth and disparity estimation

This disclosure relates to techniques for the robust usage of semantic segmentation information in image processing techniques, e.g., shallow depth of field (SDOF) renderings. Semantic segmentation may be defined as a process of creating a mask over an image, wherein pixels are segmented into a predefined set of semantic classes. Segmentations may be binary (e.g., a 'person pixel' or a 'non-person pixel') or multi-class (e.g., a pixel may be labelled as: &#x2018;person,&#x2019; &#x2018;dog,&#x2019; &#x2018;cat,&#x2019; etc.). As semantic segmentation techniques grow in accuracy and adoption, it is becoming increasingly important to develop methods of utilizing such segmentations and developing flexible techniques for integrating segmentation information into existing computer vision applications, such as synthetic SDOF renderings, to yield improved results in a wide range of image capture scenarios. In some embodiments, a refinement operation may be employed on a camera device's initial depth, disparity and/or blur estimates that leverages semantic segmentation information.

Robust use of semantic segmentation in shallow depth of field rendering

The disclosure pertains to techniques for image processing. One such technique comprises a method for image processing comprising obtaining first light information from a set of light-sensitive pixels for a scene, the pixels including phase detection (PD) pixels and non-PD pixels, generating a first PD pixel image from the first light information, the first PD pixel image having a first resolution, generating a higher resolution image from the plurality of non-PD pixels, wherein the higher resolution image has a resolution greater than the resolution of the first PD pixel image, matching a first pixel of the first PD pixel image to the higher resolution image, wherein the matching is based on a set of correlations between the first pixel and non-PD pixel within a predetermined distance of the first pixel, and determining a disparity map for an image associated with the first light information, based on the match.

Disparity estimation using sparsely-distributed phase detection pixels

This disclosure relates to techniques for synthesizing out of focus effects in digital images. Digital single-lens reflex (DSLR) cameras and other cameras having wide aperture lenses typically capture images with a shallow depth of field (SDOF). SDOF photography is often used in portrait photography, since it emphasizes the subject, while deemphasizing the background via blurring. Simulating this kind of blurring using a large depth of field (LDOF) camera may require a large amount of computational resources, i.e., in order to simulate the physical effects of using a wide aperture lens while constructing a synthetic SDOF image. However, cameras having smaller lens apertures, such as mobile phones, may not have the processing power to simulate the spreading of all background light sources in a reasonable amount of time. Thus, described herein are techniques to synthesize out-of-focus background blurring effects in a computationally-efficient manner for images captured by LDOF cameras.

Shallow depth of field rendering

Generating an image with a selected level of background blur includes capturing, by a first image capture device, a plurality of frames of a scene, wherein each of the plurality of frames has a different focus depth, obtaining a depth map of the scene, determining a target object and a background in the scene based on the depth map, determining a goal blur for the background, and selecting, for each pixel in an output image, a corresponding pixel from the focus stack.

Photo-realistic shallow depth-of-field rendering from focal stacks

Generating a focus stack, including receiving initial focus data that identifies a plurality of target depths, positioning a lens at a first position to capture a first image at a first target depth of the plurality of target depths, determining, in response to capturing the first image and prior to capturing additional images, a sharpness metric for the first image, capturing, in response to determining that the sharpness metric for the first image is an unacceptable value, a second image at a second position based on the sharpness metric, wherein the second position is not included in the plurality of target depths, determining that a sharpness metric for the second image is an acceptable value, and generating a focus stack using the second image.

Optimizing capture of focus stacks

A method to correct an autofocus operation of a digital image capture device based on an empirical evaluation of image capture metadata is disclosed. The method includes capturing an image of a scene (the image including one or more autofocus windows), obtaining an initial focus score for at least one of the image's one or more autofocus windows, obtaining image capture metadata for at least one of the one or more autofocus windows, determining a focus adjustment score for the one autofocus window based on a combination of the autofocus window's image capture metadata (wherein the focus adjustment score is indicative of the autofocus window's noise), and determining a corrected focus score for the one autofocus window based on the initial focus score and the focus adjustment score.

Growing community of inventors

San Jose, CA, United States of America

Alexander Lindskog