Apple Inc.

Michael Wish Tao

Alexander Lindskog

Geoffrey T Anneheim

Mark N Jouppi

David R Pope

Florian Ciurea

Ajay Ramesh

Alexandre Naaman

Richard David Seely

Terence N Tam

Julio Cesar Hernandez Zaragoza

Stephane S Ben Soussan

Patrick A Carroll

Bruno J Conejo

Ashwini Dwarakanath

Mark Norman Lester Jouppi

Eric Bujold

Vitanshu Sharma

Stephane Simon Rene Ben Soussan

David A Silverstein

Volker Roelke

Devices, methods, and non-transitory program storage devices are disclosed herein to perform predictive image sensor cropping operations to improve the performance of video image stabilization operations, especially for high resolution image sensors. According to some embodiments, the techniques include, for each of one or more respective images of a first plurality of images: obtaining image information corresponding to one or more images in the first plurality of images captured prior to the respective image; predicting, for the respective image, an image sensor cropping region to be read out from the first image sensor; and then reading out, into a memory, a first cropped version of the respective image comprising only the predicted image sensor cropping region for the respective image. Then, a first video may be produced based, at least in part, on the first cropped versions of the one or more respective images of the first plurality of images.

Sensor cropped video image stabilization (VIS)

This disclosure relates to techniques for generating robust depth estimations for captured images using semantic segmentation. Semantic segmentation may be defined as a process of creating a mask over an image, wherein pixels are segmented into a predefined set of semantic classes. Such segmentations may be binary (e.g., a 'person pixel' or a 'non-person pixel') or multi-class (e.g., a pixel may be labelled as: &#x2018;person,&#x2019; &#x2018;dog,&#x2019; &#x2018;cat,&#x2019; etc.). As semantic segmentation techniques grow in accuracy and adoption, it is becoming increasingly important to develop methods of utilizing such segmentations and developing flexible techniques for integrating segmentation information into existing computer vision applications, such as depth and/or disparity estimation, to yield improved results in a wide range of image capture scenarios. In some embodiments, an optimization framework may be employed to optimize a camera device's initial scene depth/disparity estimates that employs both semantic segmentation and color regularization in a robust fashion.

Robust use of semantic segmentation for depth and disparity estimation

This disclosure relates to techniques for the robust usage of semantic segmentation information in image processing techniques, e.g., shallow depth of field (SDOF) renderings. Semantic segmentation may be defined as a process of creating a mask over an image, wherein pixels are segmented into a predefined set of semantic classes. Segmentations may be binary (e.g., a 'person pixel' or a 'non-person pixel') or multi-class (e.g., a pixel may be labelled as: &#x2018;person,&#x2019; &#x2018;dog,&#x2019; &#x2018;cat,&#x2019; etc.). As semantic segmentation techniques grow in accuracy and adoption, it is becoming increasingly important to develop methods of utilizing such segmentations and developing flexible techniques for integrating segmentation information into existing computer vision applications, such as synthetic SDOF renderings, to yield improved results in a wide range of image capture scenarios. In some embodiments, a refinement operation may be employed on a camera device's initial depth, disparity and/or blur estimates that leverages semantic segmentation information.

Robust use of semantic segmentation in shallow depth of field rendering

Determining disparity includes obtaining a first image of a scene and a second image of a scene, determining correspondences between one or more pixels of the first image and one or more pixels of the second image, performing local denoising on the correspondences based on at least on a strength and direction of gradient values for the one or more pixels of the first image and the one or more pixels of the second image, and generating a disparity map based on the determined correspondences and local denoising.

Robust hardware efficient disparity estimation using total-variation L1 regularization

The disclosure pertains to techniques for image processing. One such technique comprises a method for image processing comprising obtaining first light information from a set of light-sensitive pixels for a scene, the pixels including phase detection (PD) pixels and non-PD pixels, generating a first PD pixel image from the first light information, the first PD pixel image having a first resolution, generating a higher resolution image from the plurality of non-PD pixels, wherein the higher resolution image has a resolution greater than the resolution of the first PD pixel image, matching a first pixel of the first PD pixel image to the higher resolution image, wherein the matching is based on a set of correlations between the first pixel and non-PD pixel within a predetermined distance of the first pixel, and determining a disparity map for an image associated with the first light information, based on the match.

Disparity estimation using sparsely-distributed phase detection pixels

An electronic device comprises circuitry implementing a depth map enhancer. The depth map enhancer obtains an initial depth map corresponding to a scene and an image of the scene. The depth map enhancer generates a refined depth map corresponding to the scene using an optimizer, the initial depth map and the image. The refined depth map includes estimated depth indicators corresponding to at least a first depth-information region, identified based at least in part on a first criterion, of the initial depth map. Input based on the refined depth map is provided to an image processing application.

Efficient scene depth map enhancement for low power devices

This disclosure relates to techniques for synthesizing out of focus effects in digital images. Digital single-lens reflex (DSLR) cameras and other cameras having wide aperture lenses typically capture images with a shallow depth of field (SDOF). SDOF photography is often used in portrait photography, since it emphasizes the subject, while deemphasizing the background via blurring. Simulating this kind of blurring using a large depth of field (LDOF) camera may require a large amount of computational resources, i.e., in order to simulate the physical effects of using a wide aperture lens while constructing a synthetic SDOF image. However, cameras having smaller lens apertures, such as mobile phones, may not have the processing power to simulate the spreading of all background light sources in a reasonable amount of time. Thus, described herein are techniques to synthesize out-of-focus background blurring effects in a computationally-efficient manner for images captured by LDOF cameras.

Shallow depth of field rendering

Growing community of inventors

San Jose, CA, United States of America

Michael Wish Tao