Pelican Imaging Corporation

Fotonation Limited

Fotonation Cayman Limited

Adeia Imaging LLC

Apple Inc.

Ntt Docomo, Inc.

Aptina Imaging Corporation

Kip Peli P1 LP

Adela Imaging LLC

Dan C Lelescu

Kartik Venkataraman

Gabriel Molina

Florian Ciurea

Semyon Nisenzon

Jacques Duparre

Andrew Kenneth John McMahon

Paul Gallagher

Ankit K Jain

Bedabrata Pain

Gary Lee Vondran, Jr

Robert H Mullis

Priyam Chatterjee

Thang Duong

Shree Kumar Nayar

Frank Jan Bossen

Felipe Bacim De Araujo E Silva

Manohar B Srikanth

Frank Bossen

Cheng Lu

Sebastien Eric Gilles Leprince

Samuel Yang

Ricardo Jansson Motta

David R Pope

Seung Wook Kim

Brian W Keelan

Nicolas V Scapel

Chihsin Wu

Brett D Miller

Kathrin Berkner-Cieslicki

Tobias Rick

Robert K Molholm

Pravin Rao

Robert Mullis

Gabriel D Molina

Junqing Chen

Systems and methods in accordance with embodiments of the invention are disclosed that use super-resolution (SR) processes to use information from a plurality of low resolution (LR) images captured by an array camera to produce a synthesized higher resolution image. One embodiment includes obtaining input images using the plurality of imagers, using a microprocessor to determine an initial estimate of at least a portion of a high resolution image using a plurality of pixels from the input images, and using a microprocessor to determine a high resolution image that when mapped through the forward imaging transformation matches the input images to within at least one predetermined criterion using the initial estimate of at least a portion of the high resolution image. In addition, each forward imaging transformation corresponds to the manner in which each imager in the imaging array generate the input images, and the high resolution image synthesized by the microprocessor has a resolution that is greater than any of the input images.

Systems and methods for synthesizing high resolution images using images captured by an array of independently controllable imagers

Embodiments of the invention provide a camera array imaging architecture that computes depth maps for objects within a scene captured by the cameras, and use a near-field sub-array of cameras to compute depth to near-field objects and a far-field sub-array of cameras to compute depth to far-field objects. In particular, a baseline distance between cameras in the near-field subarray is less than a baseline distance between cameras in the far-field sub-array in order to increase the accuracy of the depth map. Some embodiments provide an illumination near-IR light source for use in computing depth maps.

Multi-baseline camera array system architectures for depth augmentation in VR/AR applications

Systems and methods in accordance with embodiments of the invention are configured to render images using light field image files containing an image synthesized from light field image data and metadata describing the image that includes a depth map. One embodiment of the invention includes a processor and memory containing a rendering application and a light field image file including an encoded image, a set of low resolution images, and metadata describing the encoded image, where the metadata comprises a depth map that specifies depths from the reference viewpoint for pixels in the encoded image. In addition, the rendering application configures the processor to: locate the encoded image within the light field image file; decode the encoded image; locate the metadata within the light field image file; and post process the decoded image by modifying the pixels based on the depths indicated within the depth map and the set of low resolution images to create a rendered image.

Systems and methods for encoding image files containing depth maps stored as metadata

Novel viewpoints may be rendered using arrays of camera images with synthesized viewpoints. Different viewpoints of a scene may be captured via image sensors. Depth information may be determined for the captured viewpoints. An array of images may be generated that includes synthetic viewpoints generated from the captured viewpoints and depth information. A request to render a novel viewpoint may be received. Then novel viewpoint may be rendered using the array of images with synthetic viewpoints.

Synthesized camera arrays for rendering novel viewpoints

Various implementations disclosed herein include devices, systems, and methods that provide images for displaying a 3D appearance of a tracked object on a view dependent device. In some implementations, a set of multiple images of the object from different capture positions is provided and then adjusted such as to render the object in focus when using the adjusted images for display on the view dependent device. In some implementations, the set of multiple images is provided that includes virtual images generated using virtual image sensors that may improve the appearance of out of focus regions (e.g., outside the 3D object appearance) on the view dependent device. In some implementations, the set of multiple images of a person provides a hologram of the person (e.g., left and right eye views provide the appearance of a 3D person) where the images are adjusted to improve gaze direction of eyes of the person.

Holographic video sessions

Systems in accordance with embodiments of the invention can perform parallax detection and correction in images captured using array cameras. Due to the different viewpoints of the cameras, parallax results in variations in the position of objects within the captured images of the scene. Methods in accordance with embodiments of the invention provide an accurate account of the pixel disparity due to parallax between the different cameras in the array, so that appropriate scene-dependent geometric shifts can be applied to the pixels of the captured images when performing super-resolution processing. In a number of embodiments, generating depth estimates considers the similarity of pixels in multiple spectral channels. In certain embodiments, generating depth estimates involves generating a confidence map indicating the reliability of depth estimates.

Systems and methods for estimating depth and visibility from a reference viewpoint for pixels in a set of images captured from different viewpoints

Systems and methods for dynamically calibrating an array camera to accommodate variations in geometry that can occur throughout its operational life are disclosed. The dynamic calibration processes can include acquiring a set of images of a scene and identifying corresponding features within the images. Geometric calibration data can be used to rectify the images and determine residual vectors for the geometric calibration data at locations where corresponding features are observed. The residual vectors can then be used to determine updated geometric calibration data for the camera array. In several embodiments, the residual vectors are used to generate a residual vector calibration data field that updates the geometric calibration data. In many embodiments, the residual vectors are used to select a set of geometric calibration from amongst a number of different sets of geometric calibration data that is the best fit for the current geometry of the camera array.

Systems and methods for dynamic calibration of array cameras

A light field panorama system in which a user holding a mobile device performs a gesture to capture images of a scene from different positions. Additional information, for example position and orientation information, may also be captured. The images and information may be processed to determine metadata including the relative positions of the images and depth information for the images. The images and metadata may be stored as a light field panorama. The light field panorama may be processed by a rendering engine to render different 3D views of the scene to allow a viewer to explore the scene from different positions and angles with six degrees of freedom. Using a rendering and viewing system such as a mobile device or head-mounted display, the viewer may see behind or over objects in the scene, zoom in or out on the scene, or view different parts of the scene.

Panoramic light field capture, processing, and display

Systems and methods in accordance with embodiments of the invention actively align a lens stack array with an array of focal planes to construct an array camera module. In one embodiment, a method for actively aligning a lens stack array with a sensor that has a focal plane array includes: aligning the lens stack array relative to the sensor in an initial position; varying the spatial relationship between the lens stack array and the sensor; capturing images of a known target that has a region of interest using a plurality of active focal planes at different spatial relationships; scoring the images based on the extent to which the region of interest is focused in the images; selecting a spatial relationship between the lens stack array and the sensor based on a comparison of the scores; and forming an array camera subassembly based on the selected spatial relationship.

Systems and methods for manufacturing camera modules using active alignment of lens stack arrays and sensors

Growing community of inventors

Morgan Hill, CA, United States of America

Dan C Lelescu