Microsoft Technology Licensing, LLC

Sergio Ortiz Egea

Onur Can Akkaya

Cyrus Soli Bamji

Michael Scott Fenton

Maria Esther Pace

Alfonsus Dibianto Lunardhi

Augustine Cha

Abdelrehim Ahmed

Hung-Ming Lin

Jian Feng Gao

Venkata Satya Raghavendra Bulusu

Travis Jon Perry

Bernard Charles Kress

Mukhil Azhagan Mallaiyan Sathiaseelan

Riaz Imdad Ali

Venkata Satya Reghavendra Bulusu

Ling Zhu

Examples are disclosed that relate to motion blur corrections for time-of-flight (ToF) depth imaging. One example provides a depth camera comprising a ToF image sensor, a logic machine, and a storage machine storing instructions executable by the logic machine to receive depth image data from the ToF image sensor, the depth image data comprising phase data and active brightness (AB) data, determine a first two-dimensional (2D) AB image corresponding to a first modulation frequency, and determine a second 2D AB image corresponding to a second modulation frequency. The instructions are further executable to determine a 2D translation based upon a comparison between the first 2D AB image and the second 2D AB image, determine corrected phase data based on the 2D translation to form corrected phase data, perform phase unwrapping on the corrected phase data to obtain a three-dimensional (3D) depth image, and output the 3D depth image.

Motion correction for time-of-flight depth imaging

One example provides a computing device including a logic machine and a storage machine holding instructions executable by the logic machine to implement a depth image processing pipeline including a neural network, the neural network including an edge detecting layer. The neural network is configured to receive input of an active brightness image and receive input of one or more of real data or imaginary data of a complex depth image, the complex depth image corresponding to the active brightness image. The neural network is further configured to, at the edge detecting layer, apply one or more convolutional processes to the active brightness image to identify one or more edge pixels in the active brightness image, and at a second layer, denoise one or more of the real data or the imaginary data of the complex depth image based on the one or more edge pixels identified.

Denoising depth image data using neural networks

One example provides a computing system comprising a depth sensor comprising a plurality of pixels, and a storage machine holding instructions executable by a logic machine to, for each pixel, make K phase measurements to form a set of noisy phase measurements, determine a location at which a projection line that passes through the set of noisy phase measurements in a K-dimensional phase space passes through a lower dimensional plane, the projection line being parallel to a noise free phase evolution line, compare the location to a plurality of independent terms of a predetermined matrix of points in the lower dimensional plane, locate a corresponding set of noiseless phase orders by using a selected set of independent terms to reference a look-up table, determine a distance value for the pixel based upon the corresponding set of noiseless phase orders, and output the distance value for the pixel.

Graphical ToF phase unwrapping

Examples are disclosed relating to performing signal processing on time-of-flight sensor data using pixelwise temporal metrics. One example provides a computing system comprising a logic machine, and a storage machine holding instructions executable by the logic machine to obtain temporal phase data for a plurality of pixels as acquired by a time-of-flight image sensor, the temporal phase data comprising phase data for a plurality of light modulation frequencies, determine temporal active brightness data for the pixel, and, for each pixel of the plurality of pixels, determine a statistical metric for the temporal active brightness data. The instructions are further executable to perform phase unwrapping on the temporal phase data for the plurality of pixels to obtain a depth image, based on the statistical metric for the temporal active brightness data, perform a denoising operation on at least some pixels of the depth image, and output the depth image.

Temporal metrics for denoising depth image data

Examples are disclosed herein relating to signal processing in a time of flight (ToF) system. One example provides, a method comprising emitting, via a light source, amplitude-modulated light toward an object, acquiring, via an image sensor comprising a plurality of pixels, a plurality of image frames capturing light emitted from the light source that is reflected by the object, wherein the plurality of image frames are acquired at two or more different frequencies of the amplitude-modulated light and collectively form a multifrequency frame, and for each pixel of the multifrequency frame, determining a brightness level, applying an adaptive denoising process by setting a kernel size based on the brightness level, and performing a phase unwrapping process to determine a depth value for the pixel.

Adaptive processing in time of flight imaging

Examples are disclosed relating to performing denoising and adaptive precision control on time-of-flight sensor data using noise metrics. One example provides a computing system, comprising, a logic machine, and a storage machine holding instructions executable by the logic machine to obtain time-of-flight (ToF) image data comprising a plurality of pixels, for each pixel of the ToF image data, determine one or more noise metrics by applying a spatial kernel, segment the ToF image data based on the one or more noise metrics to obtain differently classified pixels, during a denoising phase, process pixels of a first classification differently than pixels of a second classification, after the denoising phase, determine a depth image, and output the depth image.

Spatial metrics for denoising depth image data

Examples are provided relating to recovering depth data from noisy phase data of low-signal pixels. One example provides a computing system, comprising a logic machine, and a storage machine holding instructions executable by the logic machine to process depth data by obtaining depth image data and active brightness image data for a plurality of pixels, the depth image data comprising phase data for a plurality of frequencies, and identifying low-signal pixels based at least on the active brightness image data. The instructions are further executable to apply a denoising filter to phase data of the low-signal pixels to obtain denoised phase data and not applying the denoising filter to phase data of other pixels. The instructions are further executable to, after applying the denoising filter, perform phase unwrapping on the phase data for the plurality of frequencies to obtain a depth image, and output the depth image.

Denoising depth data of low-signal pixels

Examples are provided that relate to processing depth camera data over a distributed computing system, where phase unwrapping is performed prior to denoising. One example provides a time-of-flight camera comprising a time-of-flight depth image sensor, a logic machine, a communication subsystem, and a storage machine holding instructions executable by the logic machine to process time-of-flight image data acquired by the time-of-flight depth image sensor by, prior to denoising, performing phase unwrapping pixel-wise on the time-of-flight image data to obtain coarse depth image data comprising depth values; and send the coarse depth image data and active brightness image data to a remote computing system via the communication subsystem for denoising.

Distributed depth data processing

Examples are disclosed that relate to methods and systems for determining a depth of a pixel of a depth image. In one example, a method comprises illuminating a scene with amplitude-modulated light of a plurality K of different frequencies. A sensor detects the amplitude-modulated light as reflected from the scene. A plurality K of complex signals k is formed by determining, for each frequency k of the plurality of different frequencies K, a complex signal k comprising a real portion k and an imaginary portion k. A comparison function is used to compare the plurality K of complex signals to a corresponding plurality K of modeled signals, each modeled signal k comprising a modeled real portion k and a modeled imaginary portion k. The method further comprises determining the depth of the pixel based at least on the comparison function and outputting the depth of the pixel.

Determining depth in a depth image

A method to correct a digital image to reverse the effect of signal diffusion among pixels of the digital image. For a target pixel j of the digital image, a set of signal values and a set of signal amplitudes are received, each corresponding to a set of kernel pixels i surrounding and including the target pixel j. For each kernel pixel i, a weighting coefficient is computed based on the signal amplitude of that kernel pixel i and on the signal amplitude of the target pixel j. A linear combination of signal values corresponding to the set of kernel pixels i is computed, wherein the signal value for each pixel i is weighted by the weighting coefficient corresponding to that pixel i. The linear combination is stored in volatile memory of an electronic device as a corrected signal value for the target pixel j.

Correction for pixel-to-pixel signal diffusion

Examples are disclosed herein relating to signal processing in a time-of-flight (ToF) system. One example provides, a method comprising emitting, via a light source, amplitude-modulated light toward an object, acquiring, via an image sensor comprising a plurality of pixels, a plurality of image frames capturing light emitted from the light source that is reflected by the object, wherein the plurality of image frames are acquired at two or more different frequencies of the amplitude-modulated light and collectively form a multifrequency frame, and for each pixel of the multifrequency frame, determining a brightness level, applying an adaptive denoising process by setting a kernel size based on the brightness level, and performing a phase unwrapping process to determine a depth value for the pixel.

A Near-Eye-Display (NED) devices that translates combinations of user gaze direction and predetermined facial gestures into user input instructions. The NED device includes an eye tracking system and a display that renders computer-generated images within a user's field-of-view. The eye tracking system may continually track the user's eye movements with a high degree of accuracy to identify specific computer-generated images that a user is focused on. The eye tracking system may also identify various facial gestures such as, for example, left-eye blinks and/or right-eye blinks that are performed while the specific computer-generated images are being focused on. In this way, NED devices are enabled to identify combinations of user gaze direction and predetermined facial gestures and to translate these identified combinations into user input instructions that correspond to specific computer-generated images.

Translating combinations of user gaze direction and predetermined facial gestures into user input instructions for near-eye-display (NED) devices

A camera system is configured to automatically monitor an area. Depth image(s) of the area are acquired based on active IR light emitted by the camera system and reflected from the area to a sensor array of the camera system. The depth image(s) are computer analyzed to identify a human subject. For each spectral illuminator of the camera system, spectral light image(s) of the area are acquired based on active spectral light in the spectral light sub-band of the spectral illuminator reflected from the area to the sensor array. The spectral light image(s) for the spectral illuminators are computer analyzed to identify an interaction between the human subject and an object in the area. In response to identifying the interaction between the human subject and the object in the area, an action to be performed for the object in the area is computer issued.

Growing community of inventors

San Jose, CA, United States of America

Sergio Ortiz Egea