Apple Inc.

Nokia Corporation

Nokia Technologies Oy

Other

Marius Tico

Paul M Hubel

Rolf Toft

Frank Doepke

Claus Molgaard

Todd Stephen Sachs

Anita Nariani Schulze

Jianping Zhou

Markku Kalevi Vehvilainen

Jun Hu

Ting Chen

Matti Sakari Hamalainen

Hao Sun

Koray Ozcan

Xuemei Zhang

Wei Sun

George E Williams

Alok Deshpande

Markku Vehviläinen

Wu Cheng

Morten Poulsen

Yongfang Liang

Mejdi Trimeche

Tak Shing Wong

Justin Michael Faryabi-Allen

Ralph T Brunner

Yingjun Bai

Farhan A Baqai

Marco Zuliani

Fabio Riccardi

Xiaoxing Li

Sakari Tapio Alenius

Juanqing Chen

Lachlan Pockett

Matti S Hamalainen

Olli Silven

Jani Boutellier

Tom Pylkkanen

Timo Pylvänäinen

Lech J Szumilas

Tiffany Jou Cheng

Markku K Vehvilainen

Techniques are disclosed for managing image capture and processing in a multi-camera imaging system. In such a system, a pair of cameras each may output a sequence of frames representing captured image data. The cameras' output may be synchronized to each other to cause synchronism in the image capture operations of the cameras. The system may assess image quality of frames output from the cameras and, based on the image quality, designate a pair of the frames to serve as a 'reference frame pair.' Thus, one frame from the first camera and a paired frame from the second camera will be designated as the reference frame pair. The system may adjust each reference frame in the pair using other frames from their respective cameras. The reference frames also may be processed by other operations within the system, such as image fusion.

Still image stabilization/optical image stabilization synchronization in multi-camera image capture

Devices, methods, and computer-readable media are disclosed describing an adaptive approach for image bracket selection and fusion, e.g., to generate low noise and high dynamic range (HDR) images in a wide variety of capturing conditions. An incoming image stream may be obtained from an image capture device, wherein the incoming image stream comprises a variety of differently-exposed captures, e.g., EV0 images, EV&#x2212; images, EV+ images, long exposure (or synthetic long exposure) images, EV0/EV&#x2212; image pairs, etc., which are received according to a particular pattern. When a capture request is received, a set of rules and/or a decision tree may be used to evaluate one or more capture conditions associated with the images from the incoming image stream and determine which two or more images to select for a fusion operation. A noise reduction process may optionally be performed on the selected images before (or after) the registration and fusion operations.

Unified bracketing approach for imaging

The disclosure pertains to techniques for image processing. One such technique comprises a method for image selection, comprising obtaining a sequence of images, detecting one or more objects in one or more images of the sequence of images, determining a location for each detected object in the one or more images, determining a trajectory for each detected object based on a determined location for each respective detected object in two or more images of the sequence of images, determining a trajectory waypoint score for the trajectory of each detected object, determining a set of selected images based on an aggregation of trajectory waypoint scores for each detected object in each respective image, and outputting the set of selected images for presentation.

Motion trajectory tracking for action detection

Electronic devices, methods, and program storage devices for leveraging machine learning to perform improved image fusion and/or noise reduction are disclosed. An incoming image stream may be obtained from an image capture device, wherein the incoming image stream comprises a variety of differently-exposed captures, e.g., EV0 images, EV&#x2212; images, EV+ images, long exposure images, EV0/EV&#x2212; image pairs, etc., which are received according to a particular pattern. When a capture request is received, two or more intermediate assets may be generated based on determined combinations of images from the incoming image stream, and the intermediate assets may then be fed into a neural network that has been trained to determine one or more sets of parameters to optimally fuse and/or noise reduce the intermediate assets. In some embodiments, the network may be trained to operate on levels of pyramidal decompositions of the intermediate assets independently, for increased efficiency and memory utilization.

Deep learning-based image fusion for noise reduction and high dynamic range

Disclosed herein is an apparatus. The apparatus includes a housing, electronic circuitry, and an audio-visual source tracking system. The electronic circuitry is in the housing. The audio-visual source tracking system includes a first video camera and an array of microphones. The first video camera and the array of microphones are attached to the housing. The audio-visual source tracking system is configured to receive video information from the first video camera. The audio-visual source tracking system is configured to capture audio information from the array of microphones at least partially in response to the video information. The audio-visual source tracking system might include a second video camera that is attached to the housing, wherein the first and second video cameras together estimate the beam orientation of the array of microphones.

Electronic device directional audio-video capture

Techniques to capture and fuse short- and long-exposure images of a scene from a stabilized image capture device are disclosed. More particularly, the disclosed techniques use not only individual pixel differences between co-captured short- and long-exposure images, but also the spatial structure of occluded regions in the long-exposure images (e.g., areas of the long-exposure image(s) exhibiting blur due to scene object motion). A novel device used to represent this feature of the long-exposure image is a 'spatial difference map.' Spatial difference maps may be used to identify pixels in the short- and long-exposure images for fusion and, in one embodiment, may be used to identify pixels from the short-exposure image(s) to filter post-fusion so as to reduce visual discontinuities in the output image.

Scene motion correction in fused image systems

Systems, methods, and computer readable media for adaptively selecting what portion (aka slice) of a first image (aka frame) is selected to overlap and blend with a second frame during frame capture operations are disclosed. In general, for every new frame captured in a sequence the overlap between it and the slice selected from a prior frame may be determined based, at least in part, on sensor output. If the overlap so determined is below a desired threshold, the position of the current frame's slice may be adjusted so as to provide the desired overlap.

Motion adaptive image slice selection

Foreign lighting effects, such as lens flare, are very common in natural images. In a two-camera-system, the two images may be fused together to generate one image of a better quality. However, there are frequently different foreign light patterns in the two images that form the image pair, e.g., due to the difference in lens design, sensor and position, etc. Directly fusing such pairs of images will result in non-photorealistic images, with composed foreign light patterns from both images from the image pair. This disclosure describes a general foreign light mitigation scheme to detect all kinds of foreign light region mismatches. The detected foreign light mismatch regions may be deemphasized or excluded in the fusion step, in order to create a fused image that keeps a natural-looking foreign light pattern that is close to what was seen by the user of an image capture device during an image capture preview mode.

Mismatched foreign light detection and mitigation in the image fusion of a two-camera system

Techniques to capture and fuse short- and long-exposure images of a scene from a stabilized image capture device are disclosed. More particularly, the disclosed techniques use not only individual pixel differences between co-captured short- and long-exposure images, but also the spatial structure of occluded regions in the long-exposure images (e.g., areas of the long-exposure image(s) exhibiting blur due to scene object motion). A novel device used to represent this feature of the long-exposure image is a 'spatial difference map.' Spatial difference maps may be used to identify pixels in the short-and long-exposure images for fusion and, in one embodiment, may be used to identify pixels from the short-exposure image(s) to filter post-fusion so as to reduce visual discontinuities in the output image.

Lens flare mitigation techniques determine which pixels in images of a sequence of images are likely to be pixels affected by lens flare. Once the lens flare areas of the images are determined, unwanted lens flare effects may be mitigated by various approaches, including reducing border artifacts along a seam between successive images, discarding entire images of the sequence that contain lens flare areas, and using tone-mapping to reduce the visibility of lens flare.

Flare detection and mitigation in panoramic images

Special blend operations for wide area-of-view image generation utilizing a 'floating auto exposure' scheme are described. Pixel values in the two images being stitched together are blended within a transition band around a 'seam.' identified in the overlap region between the images after changes in exposure and/or color saturation are accounted for. In some embodiments, changes in exposure and/or color saturation are accounted for through the use of one or more exposure mapping curves, the selection and use of which are based, at least in part, on a determined &#x201c;Exposure Ratio&#x201d; value, i.e., the amount that the camera's exposure settings have deviated from their initial capture settings. In other embodiments, the Exposure Ratio value is also used to determine regions along the seam where either: alpha blending, Poisson blending&#x2014;or a combination of the two&#x2014;should be used to blend in the transitional areas on each side of the seam.

Exposure mapping and dynamic thresholding for blending of multiple images using floating exposure

Growing community of inventors

Mountain View, CA, United States of America

Marius Tico