Microsoft Technology Licensing, LLC

Other

Michael Bleyer

Raymond Kirk Price

Christopher Douglas Edmonds

Denis Demandolx

Yuri Pekelny

Jian Zhao

Sudipta Narayan Sinha

Christian Markus Maekelae

Denis Claude Pierre Demandolx

Ravi Kiran Nalla

Christopher Stephen Messer

Christopher Francis Reidy

Dominik Alexander Konik

Casey L Miller

Zhiqiang Liu

Andrey Konin

Kai Zang

Evan Gregory Levine

Youding Zhu

Donald John Patrick O'Neil

Mikael Froejk Figueroa

Antonios Matakos

William Chao-Hui Hwang

Steven Nabil Bathiche

Shahram Izadi

Mark James Finocchio

Juan Vicente Esteve Balducci

Vivek Pradeep

Jonathan Gustav Paulovich

Marc Andre Leon Pollefeys

Michael Edward Samples

Michael Scott Fenton

Erroll William Wood

Christoph Rhemann

Mark Plagge

Christopher Zach

Michelle Brook

Brent Michael Wilson

Michael John Schoenberg

Matthew B Karr

Benjamin Brown

Preetika Jain

John Albert Judnich

Paul Lee

Melissa Arde Lobb

Michelle Lynn Brook

Casey Lee Miller

Denis C Demandolx

Carsten Curt Eckard Rother

Adam Garnet Kirk

A system for generating depth information from low-resolution images is configured to access a plurality of image frames capturing an environment, identify a first group of image frames from the plurality of image frames, and generate a first image comprising a first composite image of the environment using the first group of image frames as input. The first composite image has an image resolution that is higher than an image resolution of the image frames of the first group of image frames. The system is also configured to obtain a second image of the environment, where parallax exists between a capture perspective associated with the first image and a capture perspective associated with the second image. The system is also configured to generate depth information for the environment based on the first image and the second image.

Systems and methods for generating depth information from low-resolution images

Techniques for performing image alignment between a first image generated by a first camera and a second image generated by a second camera are disclosed. The image alignment is performed using staggered feature extraction in which a set of features are reused to align the second image with the first image. A first set of features are identified from within the first image, and a second set of features, which were previously detected within the second image, are accessed. The second set of features were previously used at least once to perform a previous image alignment operation. A current image alignment operation is performed by using the first set of features and by reusing the second set of features to align the first image with the second image.

Image alignment using staggered feature extraction

A system for facilitating dark current compensation by weighted filtering is configurable to (i) receive an input dark current image; (ii) generate a corrected dark current image at least by scaling pixel values of the input dark current image based upon ambient light conditions; and generate a weight map comprising a weight value for each pixel of the corrected dark current image. For each pixel of the corrected dark current image, the weight value of the weight map may be based upon a light level of the pixel of the corrected dark current image. The system is also configurable to generate an output image by utilizing the weight map to filter an input image.

Systems and methods for facilitating dark current compensation by weighted filtering

Disclosed herein are techniques for providing an illumination system that emits illumination into an environment while also enabling that system to be undetectable to certain types of external light detection systems. The system includes a single photon avalanche diode (SPAD) low light (LL) detection device and a light emitting device. The light emitting device provides illumination having a wavelength of at least 950 nanometers (nm). An intensity of the illumination is set to a level that causes the illumination to be undetectable from a determined distance away based on the roll off rate of the light. While the light emitting device is providing the illumination, the SPAD LL detection device generates an image of an environment in which the illumination is being provided.

External illumination with reduced detectability

A system for modifying a dark current image is configurable to receive an input image depicting a dark current state for one or more pixels of the input image, The dark current state for one or more pixels of the input image comprises one of: a faulty state or a non-faulty state. The system is configurable to partition the input image into a plurality of partitions and generate an updated input image by imposing at least one quantity constraint or at least one severity constraint to the plurality of partitions in association with at least one type of dark current state.

Systems and methods for selectively constraining and implementing cross talk templates for dark current images

A system for updating continuous image alignment of separate cameras identifies a previous alignment matrix associated with a previous frame pair captured at one or more previous timepoints by a reference camera and a match camera. The previous alignment matrix is based on visual correspondences in the previous frame pair. The system also identifies a current matrix associated with a current frame pair captured at one or more current timepoints by the reference camera and the match camera. The current matrix is based on visual correspondences in the current frame pair. The system also identifies a difference value associated with the reference camera or the match camera relative to the one or more previous timepoints and the one or more current timepoints. The system also generates an updated alignment matrix by using the previous alignment matrix, the current matrix, and the difference value as inputs.

Systems and methods for updating continuous image alignment of separate cameras

Techniques for generating a high resolution full color output image from lower resolution sparse color input images are disclosed. A camera generates images. The camera's sensor has a sparse Bayer pattern. While the camera is generating the images, IMU data for each image is acquired. The IMU data indicates a corresponding pose the camera was in while the camera generated each image. The images and IMU data are fed into a motion model, which performs temporal filtering on the images and uses the IMU data to generate a red-only image, a green-only image, a blue-only image, and a monochrome image. The color images are up-sampled to match the resolution of the monochrome image. A high resolution output color image is generated by combining the up-sampled images and the monochrome image.

Generate super-resolution images from sparse color information

Techniques for updating a position of overlaid image content using IMU data to reflect subsequent changes in camera positions to minimize latency effects are disclosed. A 'system camera' refers to an integrated camera that is a part of an HMD. An “external camera” is a camera that is separated from the HMD. The system camera and the external camera generate images. These images are overlaid on one another and aligned to form an overlaid image. Content from the external camera image is surrounded by a bounding element in the overlaid image. IMU data associated with both the system camera and the external camera is obtained. Based on that IMU data, an amount of movement that the system camera and/or the external camera have moved since the images were originally generated is determined. Based on that movement, the bounding element is shifted to a new position in the overlaid image.

Low motion to photon latency rapid target acquisition

A system for power efficient image acquisition is configurable to capture, using an image sensor, a plurality of partial image frames including at least a first partial image frame and a second partial image frame. The first partial image frame is captured at a first timepoint using a first subset of image sensing pixels of the plurality of image sensing pixels of the image sensor. The second partial image frame is captured at a second timepoint using a second subset of image sensing pixels of the plurality of image sensing pixels of the image sensor. The second subset of image sensing pixels includes different image sensing pixels than the first subset of image sensing pixels, and the second timepoint is temporally subsequent to the first timepoint. The system is configurable to generate a composite image frame based on the plurality of partial image frames.

Systems and methods for power efficient image acquisition using single photon avalanche diodes (SPADs)

An HMD includes a single photon avalanche diode (SPAD) array comprising a plurality of SPAD pixels. The HMD also includes a display positioned to display images for viewing by an eye of a user. The HMD also includes one or more processors and one or more hardware storage devices storing instructions that are executable by the one or more processors to configure the HMD to perform various acts associated with using the SPAD array to capture an image frame of an environment for display to the user.

SPAD array for intensity image sensing on head-mounted displays

An image sensor configured to capture imagery with mitigated noise includes a plurality of image sensing pixels arranged to form a sensor array. Each image sensing pixel of the plurality of image sensing pixels comprises an active area configured to receive photons to facilitate image capture. Each active area comprises a length and a width. For at least one image sensing pixel of the plurality of image sensing pixels, the length or the width of the active area is smaller than about 80% of a pixel pitch measurement between the at least one image sensing pixel and an adjacent image sensing pixel. A size of the active area relative to the pixel pitch measurement contributes to mitigating sensor noise for the image sensor.

Image sensing pixel configurations for reduced sensor noise

Techniques to temporally filter images via a filtering weight computation are disclosed. A first image having a first timestamp and a second image having a second timestamp are acquired. These images are generated by a camera, and the first timestamp is before the second timestamp. A motion compensation (MC) operation is performed on the first image to produce an MC image. A difference image is generated using the MC image and the second image. The difference image reflects differences in intensities that exist between the two images. A local weight map is generated based on those differences. A global weight map is generated based on certain IMU data. A final weight map is generated based on a combination of the local weight map and the global weight map. The final weight map is used to generate a filtered image.

Temporal filtering weight computation

A system for dark current compensation in SPAD imagery is configurable to capture an image frame with the SPAD array and generate a temporally filtered image by performing a temporal filtering operation using the image frame and at least one preceding image frame. The at least one preceding image frame is captured by the SPAD array at a timepoint that temporally precedes a timepoint associated with the image frame. The system is also configurable to obtain a dark current image frame. The dark current image frame includes data indicating one or more SPAD pixels of the plurality of SPAD pixels that detect an avalanche event without detecting a corresponding photon. The system is also configurable to generate a dark current compensated image by performing a subtraction operation on the temporally filtered image or the image frame based on the dark current image frame.

Systems and methods for dark current compensation in single photon avalanche diode imagery

Techniques for generating a fused enhanced image. A first image is generated using a first camera of a first modality, and a second image is generated using a second camera of a second modality. Pixels that are common between the two images are identified. Textures for the common pixels are determined. A camera characteristic, which is linked to noise, is identified. A scaling factor is applied to the textures in the first image. A first saliency is determined using the scaled textures. A second saliency is determined using the textures from the second image. An alpha map is generated and reflects edge detection weights that have been computed for each one of the common pixels based on the two saliencies. Based on the alpha map, textures are merged from the common pixels to generate the fused enhanced image.

Low light and thermal image normalization for advanced fusion

Techniques for generating a temporally filtered image designed to compensate for global motions of a camera and to compensate for local motions of an object are disclosed. A history frame and a current frame are acquired. A global motion compensation operation is performed on the history frame to reproject a pose of the history frame to match the pose of the current frame. The history frame is compared against the current frame to identify pixels that represent moving objects. For each of those pixels, an optical flow vector is computed. The optical flow vectors are then applied to those pixels to shift those pixels to new locations. These new positions, which are in the history frame, correspond to positions that were identified in the current frame. Afterwards, the current frame is temporally filtered with the history frame.

Advanced temporal low light filtering with global camera motion compensation and local object motion compensation

Growing community of inventors

Seattle, WA, United States of America

Michael Bleyer