Lytro, Inc.

Google Inc.

Other

Timothy James Knight

Colvin Pitts

Yi-Ren Ng

Kurt Akeley

Chia-Kai Liang

Yuriy Romanenko

Alex Fishman

Jason Rupert Redgrave

Albert Meixner

Brian Keith Cabral

Ofer Shacham

Daniel Finchelstein

Edward Y Chang

William R Mark

Carl Warren Craddock

Shahriar Rabii

Carl (Warren) Craddock

Don Stark

Bennett Wilburn

Fabrizio Basso

Michelle Tomasko

Jeff Kalt

Masumi Reynders

Brendan Elwood Bevensee

Kenneth Waters

Kayvon Fatahalian

Brendan Bevensee

Kurt Barton Akeley

Yuriy Aleksandrovich Romanenko

An image processor unit is described. The image processor unit includes a plurality of inputs to receive at least one input image. The image processor unit includes a plurality of outputs to provide at least one output image. The image processor unit includes a network coupled to the plurality of inputs and the plurality of outputs. The network is to couple at least one of the inputs to at least one of the outputs. The image processor unit includes an image processor circuit coupled to the network. The network to route an input image that is received at one of the inputs to the image processor circuit. The image processor circuit is to execute image signal processing program code to generate a processed output image from the input image. The network is to route the processed output image to at least one of the outputs.

Configurable and programmable image processor unit

RAW images and/or light field images may be compressed through the use of specialized techniques. The color depth of a light field image may be reduced through the use of a bit reduction algorithm such as a K-means algorithm. The image may then be retiled to group pixels of similar intensities and/or colors. The retiled image may be padded with extra pixel rows and/or pixel columns as needed, and compressed through the use of an image compression algorithm. The compressed image may be assembled with metadata pertinent to the manner in which compression was done to form a compressed image file. The compressed image file may be decompressed by following the compression method in reverse.

Compression of light field images

According to various embodiments of the present invention, the optical systems of light field capture devices are optimized so as to improve captured light field image data. Optimizing optical systems of light field capture devices can result in captured light field image data (both still and video) that is cheaper and/or easier to process. Optical systems can be optimized to yield improved quality or resolution when using cheaper processing approaches whose computational costs fit within various processing and/or resource constraints. As such, the optical systems of light field cameras can be optimized to reduce size and/or cost and/or increase the quality of such optical systems.

Optimization of optical systems for improved light field capture and manipulation

A light field data acquisition device includes optics and a light field sensor to acquire light field image data of a scene. In at least one embodiment, the light field sensor is located at a substantially fixed, predetermined distance relative to the focal point of the optics. In response to user input, the light field acquires the light field image data of the scene, and a storage device stores the acquired data. Such acquired data can subsequently be used to generate a plurality of images of the scene using different virtual focus depths.

Light field data acquisition

In various embodiments, the present invention relates to methods, systems, architectures, algorithms, designs, and user interfaces for light-field based autofocus. In response to receiving a focusing request at a camera, a light-field autofocus system captures a light-field image. An image crop that contains a region of interest and a specified border is determined. A series of refocused images are generated for the image crop at different scene depths. A focus metric is calculated for each refocused image. The scene depth of the refocused image with the best focus metric is identified as the appropriate focus. The focus motor position for the appropriate focus is selected and the focus motor is automatically driven to the selected focus motor position.

Light-field based autofocus

According to various embodiments, the system and method of the present invention process light-field image data so as to reduce color artifacts, reduce projection artifacts, and/or increase dynamic range. These techniques operate, for example, on image data affected by sensor saturation and/or microlens modulation. Flat-field images are captured and converted to modulation images, and then applied on a per-pixel basis, according to techniques described herein.

Compensating for sensor saturation and microlens modulation during light-field image processing

Light-field image data is processed in a manner that reduces projection artifacts in the presence of variation in microlens position by calibrating microlens positions. Approximate centers of disks in a light-field image are identified, and gridded calibration is performed, by fitting lines to disk centers along orthogonal directions, and then fitting a rigid grid to the light-field image. For each grid region, a corresponding disk center is computed, and a displacement vector is generated. For each grid region, the final disk center is computed as the vector sum of the grid region's geometric center and displacement vector. Calibration data, including displacement vectors, is then used in calibrating disk centers for more accurate projection of light-field images. In at least one embodiment, the imaging geometry is arranged so that disks are separated by a gap, so as to limit or eliminate ghosting.

Microlens array architecture for avoiding ghosting in projected images

In various embodiments, the present invention relates to methods, systems, architectures, algorithms, designs, and user interfaces for capturing, processing, analyzing, displaying, annotating, modifying, and/or interacting with light-field data on a light-field capture device. In at least one embodiment, the light-field capture device communicates to the user information about the scene during live-view to aid him or her in capturing light-field images that provide increased refocusing ability, increased parallax and perspective shifting ability, increased stereo disparity, and/or more dramatic post-capture effects. Additional embodiments present a standard 2D camera interface to software running on the light-field capture device to enable such software to function normally even though the device is actually capturing light-field data. Additional embodiments provide the ability to control camera optical elements to facilitate ease of composition and capture of light-field data, and/or generating a plurality of 2D video streams derived from a stream of light-field data.

Light-field processing and analysis, camera control, and user interfaces and interaction on light-field capture devices

According to various embodiments, a dolly zoom effect is generated using light field image data. The dolly zoom effect simulates an in-camera technique wherein a camera moves toward or away from the subject in such a way that the subject is kept at the same size throughout the effect. The effect causes the relative size of foreground background elements to change while foreground elements such as the subject remain the same size. By varying a parameter while projecting the light field image, the size of each object in the projection image scales depending on its relative depth as compared with the depth of the target subject, thus simulating the dolly zoom effect without any need to physically move the camera.

Generating dolly zoom effect using light field image data

According to various embodiments, multiple frames, each having image data and metadata, can be aggregated into pictures. The frames may come from different image capture devices, enabling aggregation of image data from multiple sources. Aggregation can be automatic, or it can be performed in response to user input specifying particular combinations of frames to be aggregated. In various embodiments, pictures are mutable, whereas immutability of the constituent frames is enforced. In various embodiments, certain metadata elements that are not essential to rendering can be selectively removed from frames, so as to address privacy concerns. In various embodiments, frames can be authenticated by the use of digests generated by a hash function.

Storage and transmission of pictures including multiple frames

Growing community of inventors

Palo Alto, CA, United States of America

Timothy James Knight