Disney Enterprises, Inc.

Eth Zurich (eidgenoessische Technische Hochschule Zurich)

Eth Zurich

Eth Zurich (eidgenossische Technische Hochschule Zurich)

Henning Zimmer

Alexander Sorkine Hornung

Oliver Wang

Christopher Richard Schroers

Changil Kim

Yael Pritch

Alexander Sorkine-Hornung

Mario Botsch

Markus H Gross

Federico Perazzi

Fabrice Pierre Armand Rousselle

Olga Sorkine-Hornung

David M Adler

Wojciech Krzysztof Jarosz

Olga Sorkine Hornung

Wenzel Jakob

Katja Wolff

Scott Frazier Watson

Markus Gross

Thomas Fraiser LaDuke

Aljoscha Smolic

Simon Heinzle

Anselm Grundhofer

Dumene Comploi

Max Grosse

Peter Kaufmann

Simone Meyer

David Robert Wyatt Rose

Christian Richardt

Ulrich Mueller

Particular embodiments decompose an image comprising a scene into a diffuse component and a specular component. Each of the components represent a contribution to lighting in the scene. A set of motion vectors may be extracted in order to capture motion in the scene. Finally, a final contribution of each of the components to the image may be computed based on the motion vectors.

Image decomposition and path-space motion estimation

Particular embodiments perform a light path analysis of an image comprising a scene, wherein the scene comprises at least one refractive or reflective object. The image may be decomposed based on the light path analysis into a plurality of components, each of the components representing a contribution to lighting in the scene by a different type of light interaction. For each of the components, one or more motion vectors are extracted for each of the components in order to capture motion in the scene. Finally, a final contribution of each of the components to the image is computed based on the motion vectors.

According to one implementation, a video processing system includes a computing platform having a hardware processor and a system memory storing a sample-based video denoising software code. The hardware processor executes the sample-based video denoising software code to receive a video sequence, and select a reference frame of the video sequence to denoise. For each pixel of the reference frame, the hardware processor executes the sample-based video denoising software code to map the pixel to a sample pixel in each of other frames of the video sequence, identify a first confidence value corresponding to each of the sample pixels based on the mapping, identify a second confidence value corresponding to each of the sample pixels based on the frame that includes the sample pixel, and denoise the pixel based on a weighted combination of the sample pixels determined using the first confidence values and the second confidence values.

Sample-based video denoising

According to one implementation, a video processing system includes a computing platform having a hardware processor and a system memory storing a sample-based video sharpening software code. The sample-based video sharpening software code receives a video sequence, and classifies frames of the video sequence as sharp or unsharp. For each pixel of an unsharp frame, the sample-based video sharpening software code determines a mapping of the pixel to another pixel in some or all of the sharp frames, determines a reverse mapping of each of the other pixels to the pixel, identifies a first confidence value corresponding to each of the other pixels based on the mapping, identifies a second confidence value corresponding to each of the other pixels based on the mapping and the reverse mapping, and sharpens the pixel based on a weighted combination of the other pixels determined using the first and second confidence values.

Sample-based video sharpening

Enhanced removing of noise and outliers from one or more point sets generated by image-based 3D reconstruction techniques is provided. In accordance with the disclosure, input images and corresponding depth maps can be used to remove pixels that are geometrically and/or photometrically inconsistent with the colored surface implied by the input images. This allows standard surface reconstruction methods (such as Poisson surface reconstruction) to perform less smoothing and thus achieve higher quality surfaces with more features. In some implementations, the enhanced point-cloud noise removal in accordance with the disclosure can include computing per-view depth maps, and detecting and removing noisy points and outliers from each per-view point cloud by checking if points are consistent with the surface implied by the other input views.

Point cloud noise and outlier removal for image-based 3D reconstruction

There are provided systems and methods for an interactive synchronization of multiple videos. An example system includes a memory storing a first video and a second video, the first video including first video clips and the second video including second video clips. The system further includes a processor configured to calculate a histogram based on a number of features that are similar between the first video clips and the second video clips, generate a cost matrix based on the histogram, generate a first graph that includes first nodes based on the cost matrix, compute a path through the graph using the nodes, and align the first video with the second video using the path, where the path corresponds to playback speeds for the first video and the second video.

Interactive synchronization of multiple videos

The disclosure provides an approach for image segmentation from an uncalibrated camera array. In one aspect, a segmentation application computes a pseudo depth map for each frame of a video sequence recorded with a camera array based on dense correspondences between cameras in the array. The segmentation application then fuses such pseudo depth maps computed for satellite cameras of the camera array to obtain a pseudo depth map at a central camera. Further, the segmentation application interpolates virtual green screen positions for an entire frame based on user input which provides control points and pseudo depth thresholds at the control points. The segmentation application then computes an initial segmentation based on a thresholding using the virtual green screen positions, and refines the initial segmentation by solving a binary labeling problem in a Markov random field to better align the segmentation with image edges and provide temporal coherency for the segmentation.

Video segmentation from an uncalibrated camera array

The present disclosure includes a method for modifying an input image to map to a three dimensional shape prior to forming the three dimensional shape. The method includes receiving by a processor at least two pre-forming images of a locally unique non-repeating pattern printed on an input material. The at least two pre-forming images are captured prior to the input material being formed into the three dimensional shape. The method further includes receiving by the processor at least two post-forming images of the pattern printed on the input material, wherein the post-forming images are captured after the input material has been formed into the three dimensional shape and analyzing by the processor the at least pre-forming images and the at least two post-forming images to determine a translation table.

Mapping for three dimensional surfaces

Systems and methods for generating a panoramic video from unstructured camera arrays. The systems and methods are configured to statically align corresponding image-frames of respective input video streams, warp the aligned image-frames according to a warping-order, and relax the warped image-frames thereby generating a temporally coherent panoramic video. Methods according to embodiments this invention utilize a new parallax-warping-error metric that is devised to capture structural differences created by parallax artifacts. The parallax-warping-error metric is effective in finding an optimal warping-order and in driving the warping process, resulting in a panoramic video with minimal parallax artifacts.

Panoramic video from unstructured camera arrays with globally consistent parallax removal

The disclosure provides an approach for estimating depth in a scene. According to one aspect, regions where the depth estimation is expected to perform well may first be identified in full-resolution epipolar-plane images (EPIs) generated from a plurality of images of the scene. Depth estimates for EPI-pixels with high edge confidence are determined by testing a number of discrete depth hypotheses and picking depths that lead to highest color density of sampled EPI-pixels. The depth estimate may also be propagated throughout the EPIs. This process of depth estimation and propagation may be iterated until all EPI-pixels with high edge confidence have been processed, and all EPIs may also be processed in this manner. The EPIs are then iteratively downsampled to coarser resolutions, at which edge confidence for EPI-pixels not yet processed are determined, depth estimates of EPI-pixels with high edge confidence made, and depth estimates propagated throughout the EPIs.

Scene reconstruction from high spatio-angular resolution light fields

Interpolating frames of a video may provide a technique for one or more of frame rate conversion, temporal upsampling for generating slow motion video, image morphing, virtual view synthesis, and/or other video applications. A system may be configured to interpolated frames of a video by leveraging frequency domain representations of individual frames. The frequency domain representations may be decomposed into set of discrete functions that make up the frequency domain representations. Corresponding functions from sets of functions associated with frames with which an interpolated frame is to be determined may be identified. Phase differences between corresponding functions may be determined. Interpolated functions between the corresponding functions may be determined based on the determined phased differences. Information describing spatial domain representations of interpolated frames may be determined based on the interpolated functions.

Systems and methods for interpolating frames of a video

Systems and methods to generate stereoscopic panoramas obtain images based on captured images. The obtained images may be processed and/or preprocessed, for example to compensate for perspective distortion caused by the non-ideal camera orientation during capturing, to reduce vertical parallax, to align adjacent images, and/or to reduce rotational and/or positional drift between adjacent images. The obtained images may be used for interpolating virtual in-between images on the fly to reduce visible artifacts in the resulting panorama. Obtained and/or interpolated images (or image fragments) may be stitched together to form a stereoscopic panorama.

Stereoscopic panoramas

A device and method incorporates features of a temporal contrast sensor with a camera sensor in an imager. The method includes registering the camera sensor with the temporal contrast sensor as a function of a calibration target. The method includes receiving camera sensor data from the camera sensor and temporal contrast sensor data from the temporal contrast sensor. The method includes generating a plurality of images as a function of incorporating the temporal contrast sensor data with the camera sensor data.

Device and method for calibrating a temporal contrast sensor with a frame-based camera sensor

Methods and systems for generating stereoscopic content with granular control over binocular disparity based on multi-perspective imaging from representations of light fields are provided. A reference image, a three-dimensional ('3D') representation of a light field corresponding to the reference image, and a goal disparity image that indicates a goal binocular disparity for one or more pixels of the reference image may be received. For each pixel of an output image corresponding to the reference image, a point within the light field that is a closest match for the goal binocular disparity of a corresponding pixel of the goal disparity image may be determined. A stereoscopic image pair including the reference image and the output image may be generated.

Growing community of inventors

Zurich, Switzerland

Henning Zimmer