Disney Enterprises, Inc.

Eth Zurich (eidgenoessische Technische Hochschule Zurich)

Simon Heinzle

Wojciech Matusik

Alexander Sorkine Hornung

Aljosa Aleksej Andrej Smolic

Nicola V Ranieri

Pierre Greisen

Markus H Gross

Lanny Starkes Smoot

Quinn Smithwick

Markus Gross

Aljoscha Smolic

Peter Charles Barnum

Changil Kim

Yael Pritch

Michael Schaffner

Oliver Wang

Tunc Ozan Aydin

Henning Zimmer

Anselm Grundhofer

Daniel James Reetz

Manuel Lang

Alexandre Chapiro

Jan Kautz

Marcus Gross

James Tompkin

Frank Kagan Gürkaynak

Steven Charles Poulakos

Techniques of depth modification for display applications are disclosed. A multiscopic video frame, including at least first and second images depicting a scene, is received. Depth information pertaining to the multiscopic video frame is received. A depth mapping function is determined based on one or more image characteristics of the multiscopic video frame, other than depth. The depth information is modified based on the depth mapping function. At least a third image is generated based on the modified depth information and based further on at least one of the first and second images, where the third image is output for display.

Depth modification for display applications

Methods and systems for generating stereoscopic content with granular control over binocular disparity based on multi-perspective imaging from representations of light fields are provided. The stereoscopic content is computed as piecewise continuous cuts through a representation of a light field, minimizing an energy reflecting prescribed parameters such as depth budget, maximum binocular disparity gradient, desired stereoscopic baseline. The methods and systems may be used for efficient and flexible stereoscopic post-processing, such as reducing excessive binocular disparity while preserving perceived depth or retargeting of already captured scenes to various view settings. Moreover, such methods and systems are highly useful for content creation in the context of multi-view autostereoscopic displays and provide a novel conceptual approach to stereoscopic image processing and post-production.

Multi-perspective stereoscopy from light fields

A multi-planar plenoptic display assembly is provided that includes multiple spatially-varying light emitting and light modulating planes. The display assembly includes at least one light emitting device and may include, but does not require, a modulating device used in conjunction according to display methods taught herein to display light field data. A display assembly controller may be used to render a light field with depth into a multi-planar plenoptic display assembly by assigning decomposed portions of the light field to the display assembly for display or presentation by differing ones of the emitting elements and by operating a modulating device to provide a parallax barrier. In one embodiment, a dynamic parallax barrier and a number of bi-state screens. Another embodiment uses a beam splitter to co-locate two pairs of autostereoscopic displays each including a projector projecting 3D content, a parallax barrier, and an emissive/projector element.

Mutli-layer plenoptic displays that combine multiple emissive and light modulating planes

A lenticular display device that is effective in increasing both perceived angular resolution and spatial resolution. These desirable results are achieved by modifying the lenslet array to better match the content of a given light field. An optimization algorithm or method (which may be implemented with software run on a computing device) is provided that analyzes an input light field and computes an optimal lenslet size, shape, and arrangement of sets of lenslets across the width of the array to better (or even best) match the input light field given a set of output parameters. The resulting lenticular display device (or print) shows higher detail and smoother motion parallax compared with fixed-size lens arrays. The usefulness of these content-adaptive lenticular prints has been demonstrated or proven using rendered simulations, by generating 3D-printed lens arrays according to the present description, and with user studies.

Content-adaptive lenticular prints

Approaches are described for generating a multiview autostereoscopic image from a stereo three-dimensional input image pair. A stereo to multiview rendering system receives a stereo three-dimensional image including a left image and a right image at a first view position and a second view position, respectively. The system generates a first input warp and a second input warp that maps the left image and the right image to a third and fourth positions, respectively, where the third and fourth positions lie between the first and second view positions. The system generates a plurality of output warps based on the first warp and the second warp. The system resamples each output warp in the plurality of output warps to create a plurality of partial output images. The system interleaves the plurality of partial output images to generate a composite output image.

Efficient stereo to multiview rendering using interleaved rendering

A device and method incorporates features of a temporal contrast sensor with a camera sensor in an imager. The method includes registering the camera sensor with the temporal contrast sensor as a function of a calibration target. The method includes receiving camera sensor data from the camera sensor and temporal contrast sensor data from the temporal contrast sensor. The method includes generating a plurality of images as a function of incorporating the temporal contrast sensor data with the camera sensor data.

Device and method for calibrating a temporal contrast sensor with a frame-based camera sensor

A closed-loop control system for stereoscopic video capture is provided. At least two motorized lenses are positioned in accordance with specified parameters to capture spatially-disparate images of a scene. The motorized lenses focus light on a corresponding one of the at least two sensors, which generate image streams. One or more processors execute instructions to provide a stream analyzer and a control module. The stream analyzer receives the image streams from the sensors and analyzes the image streams and the specified parameters in real time; the stream analyzer then modifies the image streams and generates metadata. The control module then receives and analyzes the image streams and metadata and transmits updated parameters to a control mechanism that is coupled to the at least two motorized lenses. The control mechanism then modifies operation of the at least two motorized lenses in real time in accordance with the updated parameters.

Computational stereoscopic camera system

Techniques are disclosed for rendering images. The techniques include receiving an input image associated with a source space, the input image comprising a plurality of source pixels, and applying an adaptive transformation to a source pixel, where the adaptive transformation maps the source pixel to a target space associated with an output image comprising a plurality of target pixels. The techniques further include determining a target pixel affected by the source pixel based on the adaptive transformation. The techniques further include writing the transformed source pixel into a location in the output image associated with the target pixel.

Efficient EWA video rendering

Techniques are disclosed for retargeting images. The techniques include receiving one or more input images, computing a two-dimensional saliency map based on the input images in order to determine one or more visually important features associated with the input images, projecting the saliency map horizontally and vertically to create at least one of a horizontal and vertical saliency profile, and scaling at least one of the horizontal and vertical saliency profiles. The techniques further include creating an output image based on the scaled saliency profiles. Low saliency areas are scaled non-uniformly while high saliency areas are scaled uniformly. Temporal stability is achieved by filtering the horizontal resampling pattern and the vertical resampling pattern over time. Image retargeting is achieved with greater efficiency and lower compute power, resulting in a retargeting architecture that may be implemented in a circuit suitable for mobile applications such as mobile phones and tablet computers.

Video retargeting using content-dependent scaling vectors

A multi-planar plenoptic display assembly with multiple spatially-varying light emitting and modulating planes. The display assembly includes at least one light emitting device and may include a modulating device used in conjunction according to display methods taught herein to display light field data. A display assembly controller may be used to render a light field with depth into a multi-planar plenoptic display assembly by assigning decomposed portions of the light field to the display assembly for display or presentation by differing ones of the emitting elements and by operating a modulating device to provide a parallax barrier. In one embodiment, a projector is used with bi-state screens. In another embodiment, two automultiscopic displays (either parallax barrier or lenticular lenses) are overlaid with a beam splitter. In a further embodiment, an oscillating mirror is used to temporally and optically move one automultiscopic layer (either parallax barrier or lenses) through space.

Multi-layer plenoptic displays that combine multiple emissive and light modulating planes

A method for performing light-based calibration of optics with caustic surfaces. The method includes mapping a light detecting device to a programmable light source. Then, the method includes operating a calibration light source to direct light onto one or more caustic surfaces of an optical assembly, e.g., an assembly of one or more lenses, facets, lenticules, and lenslets. The method may then involve, with the light detecting device, capturing an image of a projection surface of the optical assembly, which is opposite the one or more caustic surfaces in the optical assembly, as the projection surface is illuminated by the light from the light source. Further, the method includes processing the captured image, along with the mapping of the light detecting device to the programmable light source, to generate a calibration map of the optical assembly including the caustic surfaces.

Growing community of inventors

Zurich, Switzerland

Simon Heinzle