Lytro, Inc.

Leland Stanford Junior University

Google Inc.

Microsoft Technology Licensing, LLC

Microsoft Corporation

Yi-Ren Ng

Colvin Pitts

Mark A Horowitz

Timothy James Knight

Patrick M Hanrahan

Chia-Kai Liang

Marc Stewart Levoy

Kurt Akeley

Steven David Oliver

Graham Butler Myhre

Marc A Levoy

Ravi Kiran Nalla

Jerome Chandra Bhat

William D Houck, Ii

Kayvon Fatahalian

Tony Yip Pang Poon

Bennett Wilburn

Brandon Elliott Merle Clarke

David John Evans, V

Brian Keith Cabral

John Michael Snyder

Peter-Pike Johannes Sloan

Yuriy Romanenko

Alex Fishman

Kira Wampler

Kristen Berman

Eric Cheng

Jason Luke Wilson

Kenneth Waters

Michael Adkins Knott

Carl (Warren) Craddock

Jeff Kalt

Mugur Marculescu

Partrick M Hanrahan

A video refocusing system operates in connection with refocusable video data, information, images and/or frames, which may be light field video data, information, images and/or frames, that may be focused and/or refocused after acquisition or recording. A video acquisition device acquires first refocusable light field video data of a scene, stores first refocusable video data representative of the first refocusable light field video data, acquires second refocusable light field video data of the scene after acquiring the first refocusable light field video data, determines a first virtual focus parameter (such as a virtual focus depth) using the second refocusable light field video data, generates first video data using the stored first refocusable video data and the first virtual focus parameter, wherein the first video data includes a focus depth that is different from an optical focus depth of the first refocusable light field video data, and outputs the first video data.

Video refocusing

An image capture device, such as a camera, has multiple modes including a light field image capture mode, a conventional 2D image capture mode, and at least one intermediate image capture mode. By changing position and/or properties of the microlens array (MLA) in front of the image sensor, changes in 2D spatial resolution and angular resolution can be attained. In at least one embodiment, such changes can be performed in a continuous manner, allowing a continuum of intermediate modes to be attained.

Image capture device having light field image capture mode, 2D image capture mode, and intermediate capture mode

According to various embodiments of the invention, a system and method are provided for enabling interaction with, manipulation of, and control of depth-assigned content in depth-enhanced pictures, such as virtual reality images. Depth-assigned content can be assigned to a specified depth value. When a depth-enhanced picture is refocused at a focus depth substantially different from the specified assigned depth value, the depth-assigned content may be omitted, grayed out, blurred, or otherwise visually distinguished. In this manner, content associated with an in-focus image element can be visually distinguished from content associated with an out-of-focus image element. For example, in at least one embodiment, depth-assigned content is visible only when an image element associated with the content is in focus (or nearly in focus). According to various embodiments of the invention, many different types of interactions are facilitated among depth-assigned content, depth-enhanced virtual reality images, and other content.

Depth-assigned content for depth-enhanced virtual reality images

Various approaches to imaging involve selecting directional and spatial resolution. According to an example embodiment, images are computed using an imaging arrangement to facilitate selective directional and spatial aspects of the detection and processing of light data. Light passed through a main lens is directed to photosensors via a plurality of microlenses. The separation between the microlenses and photosensors is set to facilitate directional and/or spatial resolution in recorded light data, and facilitating refocusing power and/or image resolution in images computed from the recorded light data. In one implementation, the separation is varied between zero and one focal length of the microlenses to respectively facilitate spatial and directional resolution (with increasing directional resolution, hence refocusing power, as the separation approaches one focal length).

Variable imaging arrangements and methods therefor

According to various embodiments of the invention, a system and method are provided for enabling interaction with, manipulation of, and control of depth-assigned content in depth-enhanced pictures. Depth-assigned content can be assigned to a specified depth value. When a depth-enhanced picture is refocused at a focus depth substantially different from the specified assigned depth value, the depth-assigned content may be omitted, grayed out, blurred, or otherwise visually distinguished. In this manner, content associated with an in-focus image element can be visually distinguished from content associated with an out-of-focus image element. For example, in at least one embodiment, depth-assigned content is visible only when an image element associated with the content is in focus (or nearly in focus). According to various embodiments of the invention, many different types of interactions are facilitated among depth-assigned content, depth-enhanced pictures, and other content.

Depth-assigned content for depth-enhanced pictures

A method is performed to refocus a digital photographic image comprising a plurality of pixels. In the method, a set of images is computed corresponding to the digital photographic image and focused at different depths. Refocus depths for at least a subset of the pixels are identified and stored in a look-up table. At least a portion of the digital photographic image is refocused at a desired refocus depth determined from the look-up table.

Interactive refocusing of electronic images

Image data is processed to facilitate focusing and/or optical correction. According to an example embodiment of the present invention, an imaging arrangement collects light data corresponding to light passing through a particular focal plane. The light data is collected using an approach that facilitates the determination of the direction from which various portions of the light incident upon a portion of the focal plane emanate from. Using this directional information in connection with value of the light as detected by photosensors, an image represented by the light is selectively focused and/or corrected.

Imaging arrangements and methods therefor

A light field data acquisition device includes optics and a light field sensor to acquire light field image data of a scene. In at least one embodiment, the light field sensor is located at a substantially fixed, predetermined distance relative to the focal point of the optics. In response to user input, the light field acquires the light field image data of the scene, and a storage device stores the acquired data. Such acquired data can subsequently be used to generate a plurality of images of the scene using different virtual focus depths.

Light field data acquisition

An optical assembly includes a solid spacing layer between a plenoptic microlens array (MLA) and a pixel-level MLA, avoiding the need for an air gap. Such an assembly, and systems and methods for manufacturing same, can yield improved reliability and efficiency of production, and can avoid many of the problems associated with prior art approaches. In at least one embodiment, the plenoptic MLA, the spacing layer, and the pixel-level MLA are created from optically transmissive polymer(s) deposited on the photosensor array and shaped using photolithographic techniques. Such an approach improves precision in placement and dimensions, and avoids other problems associated with conventional polymer-on-glass architectures. Further variations and techniques are described.

Optical assembly including plenoptic microlens array

A dual-mode light field camera or plenoptic camera is enabled to perform both 3D light field imaging and conventional high-resolution 2D imaging, depending on the selected mode. In particular, an active system is provided that enables the microlenses to be optically or effectively turned on or turned off, allowing the camera to selectively operate as a 2D imaging camera or a 3D light field camera.

Light field image capture device having 2D image capture mode

According to various embodiments, the system and method disclosed herein facilitate the design of plenoptic camera lens systems to enhance camera resolution. A first configuration for the plenoptic camera may first be selected, with a first plurality of variables that define attributes of the plenoptic camera. The attributes may include a main lens attribute of a main lens of the plenoptic camera and/or a phase mask attribute of a phase mask of the plenoptic camera. A merit function may be applied by simulating receipt of light through the main lens and the plurality of microlenses of the first configuration to calculate a first merit function value. The main lens attribute and/or the phase mask attribute may be iteratively perturbed, and the merit function may be re-applied. An optimal set of variables may be identified by comparing results of successive applications of the merit function.

Plenoptic camera resolution

According to various embodiments, the system and method of the present invention process light-field image data so as to reduce color artifacts, reduce projection artifacts, and/or increase dynamic range. These techniques operate, for example, on image data affected by sensor saturation and/or microlens modulation. Flat-field images are captured and converted to modulation images, and then applied on a per-pixel basis, according to techniques described herein.

Compensating for sensor saturation and microlens modulation during light-field image processing

Light-field image data is processed in a manner that reduces projection artifacts in the presence of variation in microlens position by calibrating microlens positions. Approximate centers of disks in a light-field image are identified, and gridded calibration is performed, by fitting lines to disk centers along orthogonal directions, and then fitting a rigid grid to the light-field image. For each grid region, a corresponding disk center is computed, and a displacement vector is generated. For each grid region, the final disk center is computed as the vector sum of the grid region's geometric center and displacement vector. Calibration data, including displacement vectors, is then used in calibrating disk centers for more accurate projection of light-field images. In at least one embodiment, the imaging geometry is arranged so that disks are separated by a gap, so as to limit or eliminate ghosting.

Microlens array architecture for avoiding ghosting in projected images

Growing community of inventors

Palo Alto, CA, United States of America

Yi-Ren Ng