Microsoft Technology Licensing, LLC

Google Inc.

Mirko Schmidt

John Peter Godbaz

Cyrus Soli Bamji

Travis Perry

Ivan Dryanovski

Michael Anthony Hall

Onur Gonen Guleryuz

Eyal Krupka

Jonathan Tilton Barron

Shahram Izadi

Neal Wadhwa

Adarsh Prakash Murthy Kowdle

Michael Scott Fenton

Sean Ryan Francesco Fanello

Vladimir Tankovich

Christoph Rhemann

Daniel Freedman

Ambrus Csaszar

Michael John Schoenberg

Konstantine Nicholas John Tsotsos

Sameh Khamis

Vivek Verma

Eric Turner

Ido Leichter

Julien Valentin

Yoni Smolin

Mira Leung

Maksym Dzitsiuk

Jose Pascoal

João Manuel Castro Afonso

A handheld user device includes a monocular camera to capture a feed of images of a local scene and a processor to select, from the feed, a keyframe and perform, for a first image from the feed, stereo matching using the first image, the keyframe, and a relative pose based on a pose associated with the first image and a pose associated with the keyframe to generate a sparse disparity map representing disparities between the first image and the keyframe. The processor further is to determine a dense depth map from the disparity map using a bilateral solver algorithm, and process a viewfinder image generated from a second image of the feed with occlusion rendering based on the depth map to incorporate one or more virtual objects into the viewfinder image to generate an AR viewfinder image. Further, the processor is to provide the AR viewfinder image for display.

Depth from motion for augmented reality for handheld user devices

An HMD adjusts adjusting depth information based on detected motion of the system. The HMD includes a depth camera that collects depth data for objects in the local environment of the HMD. The HMD further includes an inertial measurement unit (IMU) including non-visual motion sensors such as one or more accelerometers, gyroscopes, and the like. The HMD adjusts the received depth information based on motion data provided by the IMU, thereby improving the accuracy of the depth information, and in turn reducing visual artifacts that can result from inaccuracies in the depth information.

Depth data adjustment based on non-visual pose data

The time-of-flight system disclosed herein includes a frequency unwrapping module configured to generate an input phase vector with M phases corresponding to M sampled signals from an object, determine an M&#x2212;1 dimensional vector of transformed phase values by applying a transformation matrix (T) to the input phase vector, determine an M&#x2212;1 dimensional vector of rounded transformed phase values by rounding the transformed phase values to a nearest integer, and determine a one dimensional lookup table (LUT) index value by transforming the M&#x2212;1 dimensional rounded transformed phase values. The index value is input into the one dimensional LUT to determine a range of the object.

Multi-frequency unwrapping

A method for facilitating removal of multipath signal interference from light data can comprise illuminating, with an illumination unit, a target with a light source. The illumination unit can be configured to project a high spatial-frequency pattern onto the target in such a way as to redistribute spectral energy to higher frequencies. The method can also comprise acquiring, with a sensor unit, reflected light data reflected from the target. The reflected light data can comprise an array of spatial domain information received from light reflected by the target. Further, the method can comprise processing, with the one or more processors, the reflected light data. The processing applies a high-pass filter within the spatial domain to the reflected light data.

Multipath signal removal in time-of-flight camera apparatus

An electronic device includes a time of flight (ToF) camera and one or more processors. The ToF camera captures raw depth images. The processors determine a depth frame and an amplitude frame from the raw depth images. The depth frame comprises an array of pixels, each pixel having a depth value. The amplitude frame comprises an array of pixels, each pixel having an amplitude energy value. The processors determine a first energy threshold value based on the amplitude energy values of the array of pixels of the amplitude frame and determine, for the depth value of a first pixel of the depth frame, a confidence value representing a corresponding validity of a depth represented by the depth value, based on a comparison of the amplitude energy value of a corresponding first pixel of the amplitude frame to the first energy threshold value.

System for depth data filtering based on amplitude energy values

A method for facilitating removal of specular reflection noise from light data can include illuminating, using an illumination unit, a target with a light source. The illumination unit is configured to project light with a spatial light pattern onto the target. The method can also include acquiring, with a sensor unit, light data that is reflected from the target. The light data may comprise a directly reflected spatial light pattern and a specular reflected spatial light pattern. The directly reflected spatial light pattern and the specular reflected spatial light pattern comprise at least one spatial distinction that distinguishes the directly reflected spatial light pattern from the specular reflected spatial light pattern. The method can further comprise processing the light data to distinguish the directly reflected spatial light pattern from the specular reflected spatial light pattern based upon the at least one spatial distinction.

Specular reflection removal in time-of-flight camera apparatus

A method for operating a time of flight (TOF) depth camera is provided. The method includes, using an image processing module, interpolating an updated timing delay calibration for each of a plurality of pixel sensors based at least on an updated set of modulation frequency and duty cycle calibration combinations received by the image processing module, the plurality of pixel sensors coupled to a timing clock, and receiving light generated by a light source and reflected in a 3-dimensional environment, the updated set of modulation frequency and duty cycle calibration combinations replacing the corresponding factory-preloaded timing delay calibrations. The method further includes applying the updated timing delay calibrations to pixel data corresponding to each of the plurality of the pixel sensors to generate a depth map of the 3-dimensional environment.

Time of flight depth camera

Fast general multipath correction in time of flight imaging is described, for example, to obtain accurate depth maps at frame rate from a time of flight camera. In various embodiments accurate depth maps are calculated by looking up corrected depth values stored in a look up table. In various embodiments the corrected depth values are highly accurate as they take into account three or more possible light ray paths between the camera and a surface in a scene being imaged. In an example accurate depth maps are computed at a frame rate of a time of flight camera. In an example accurate depth maps are computed in less than 30 milliseconds for an image having over 200,000 pixels using a standard CPU.

Fast general multipath correction in time-of-flight imaging

Growing community of inventors

San Francisco, CA, United States of America

Mirko Schmidt