Microsoft Technology Licensing, LLC

Simon Winder

Richard S Szeliski

Sing Bing Kang

Matthew Uyttendaele

Charles Lawrence Zitnick, Iii

Kestutis Patiejunas

David Eraker

Adam G Kirk

Matthew Alun Brown

Yaron Eshet

Charles Lawrence Zitnick Iii

Matthew T Uyttendaele

Charles Zitnick Iii

Methods and systems for generating a depth map are provided. The method includes projecting an infrared (IR) dot pattern onto a scene. The method also includes capturing stereo images from each of two or more synchronized IR cameras, detecting a number of dots within the stereo images, computing a number of feature descriptors for the dots in the stereo images, and computing a disparity map between the stereo images. The method further includes generating a depth map for the scene using the disparity map.

Generating a depth map

A process for compressing and decompressing non-keyframes in sequential sets of contemporaneous video frames making up multiple video streams where the video frames in a set depict substantially the same scene from different viewpoints. Each set of contemporaneous video frames has a plurality frames designated as keyframes with the remaining being non-keyframes. In one embodiment, the non-keyframes are compressed using a multi-directional spatial prediction technique. In another embodiment, the non-keyframes of each set of contemporaneous video frames are compressed using a combined chaining and spatial prediction compression technique. The spatial prediction compression technique employed can be a single direction technique where just one reference frame, and so one chain, is used to predict each non-keyframe, or it can be a multi-directional technique where two or more reference frames, and so chains, are used to predict each non-keyframe.

Compressing and decompressing multiple, layered, video streams employing multi-directional spatial encoding

Image descriptor quantization technique embodiments are presented which quantize an image descriptor defined by a vector of number elements. This is generally accomplished by lowering the number of bits per number element to a prescribed degree. The resulting quantized image descriptor exhibits minimal loss of matching reliability while at the same time reducing the amount of storage space needed to store the descriptor in a database. Lowering the number of bits per number element also allows for increased matching speed.

Image descriptor quantization

An interest point detection technique is presented. More particularly, for each of possibly multiple image pyramid resolutions, a cornerness image is generated. One or more potential interest point locations are identified in the cornerness image. This involves finding locations associated with a pixel that exhibits a higher corner strength value than pixels in a prescribed-sized surrounding pixel neighborhood. The potential interest point locations are then clustered to identify groups that likely derive from a same 2D structure. Potential interest point locations in one or more of the identified groups are respectively combined to produce a single location that represents the combined group. The representative location of each group having one is then designated as an interest point. An optional location refinement can also be implemented.

Interest point detection

Multi-pass image resampling technique embodiments are presented that employ a series of one-dimensional filtering, resampling, and shearing stages to achieve good efficiency while maintaining high visual fidelity. In one embodiment, high-quality (multi-tap) image filtering is used inside each one-dimensional resampling stage. Because each stage only uses one-dimensional filtering, the overall computation efficiency is very good and amenable to graphics processing unit (GPU) implementation using pixel shaders. This embodiment also upsamples the image before shearing steps in a direction orthogonal to the shearing to prevent aliasing, and then downsamples the image to its final size with high-quality low-pass filtering. This ensures that none of the stages causes excessive blurring or aliasing.

Multi-pass image resampling

A local image descriptor generation technique that produces a descriptor for an image patch is presented. The technique generally involves smoothing the pixels of the image patch, followed by employing a transformation to produce a transform vector for each of a set of sample points spaced across the image patch. The transform vectors are weighted and spatially accumulated to produce a prescribed number of linearly summed vectors. The linearly summed vectors are concatenated to form a raw local image descriptor, which is normalized to produce a finalized descriptor for the image patch.

Local image descriptors

A system and process for compressing and decompressing multiple video streams depicting substantially the same dynamic scene from different viewpoints that from a grid of viewpoints. Each frame in each contemporaneous set of video frames of the multiple streams is represented by at least a two layers&#x2014;a main layer and a boundary layer. Compression of the main layers involves first designating one or more of these layers in each set of contemporaneous frames as keyframes. For each set of contemporaneous frames in time sequence order, the main layer of each keyframe is compressed using an inter-frame compression technique. In addition, the main layer of each non-keyframe within the frame set under consideration is compressed using a spatial prediction compression technique. Finally, the boundary layers of each frame in the current frame set are each compressed using an intra-frame compression technique. Decompression is generally the reverse of the compression process.

System and process for compressing and decompressing multiple, layered, video streams of a scene captured from different viewpoints forming a grid using spatial and temporal encoding

A system and process for compressing and decompressing multiple video streams depicting substantially the same dynamic scene from different viewpoints. Each frame in each contemporaneous set of video frames of the multiple streams is represented by at least a two layers&#x2014;a main layer and a boundary layer. Compression of the main layers involves first designating one or more of these layers in each set of contemporaneous frames as keyframes. For each set of contemporaneous frames in time sequence order, the main layer of each keyframe is compressed using an inter-frame compression technique. In addition, the main layer of each non-keyframe within the frame set under consideration is compressed using a spatial prediction compression technique. Finally, the boundary layers of each frame in the current frame set are each compressed using an intra-frame compression technique. Decompression is generally the reverse of the compression process.

System and process for compressing and decompressing multiple, layered, video streams employing spatial and temporal encoding

A system and process for generating High Dynamic Range (HDR) video is presented which involves first capturing a video image sequence while varying the exposure so as to alternate between frames having a shorter and longer exposure. The exposure for each frame is set prior to it being captured as a function of the pixel brightness distribution in preceding frames. Next, for each frame of the video, the corresponding pixels between the frame under consideration and both preceding and subsequent frames are identified. For each corresponding pixel set, at least one pixel is identified as representing a trustworthy pixel. The pixel color information associated with the trustworthy pixels is then employed to compute a radiance value for each pixel set to form a radiance map. A tone mapping procedure can then be performed to convert the radiance map into an 8-bit representation of the HDR frame.

System and process for generating high dynamic range video

A system and process for computing a 3D reconstruction of a scene from multiple images thereof, which is based on a color segmentation-based approach, is presented. First, each image is independently segmented. Second, an initial disparity space distribution (DSD) is computed for each segment, using the assumption that all pixels within a segment have the same disparity. Next, each segment's DSD is refined using neighboring segments and its projection into other images. The assumption that each segment has a single disparity is then relaxed during a disparity smoothing stage. The result is a disparity map for each image, which in turn can be used to compute a per pixel depth map if the reconstruction application calls for it.

Color segmentation-based stereo 3D reconstruction system and process employing overlapping images of a scene captured from viewpoints forming either a line or a grid

Color segmentation-based stereo 3D reconstruction system and process

A system and process for generating, and then rendering and displaying, an interactive viewpoint video in which a user can watch a dynamic scene while manipulating (freezing, slowing down, or reversing) time and changing the viewpoint at will. In general, the interactive viewpoint video is generated using a small number of cameras to capture multiple video streams. A multi-view 3D reconstruction and matting technique is employed to create a layered representation of the video frames that enables both efficient compression and interactive playback of the captured dynamic scene, while at the same time allowing for real-time rendering.

Growing community of inventors

Seattle, WA, United States of America

Simon Winder