Microsoft Technology Licensing, LLC

John Peter Godbaz

Michael Scott Fenton

Mirko Schmidt

Cyrus Soli Bamji

Minseok Oh

Travis Perry

Andrew Dean Payne

Rui Jin

Zhanping Xu

Andrew David Daniel

Brock McCray Roland

Michael S Fenton

Michael Fenton

Techniques for using a sensor to perform laser signal decoding are disclosed. The sensor may be a global shutter sensor or a rolling shutter sensor. The sensor generates a first set of images while operating in a first mode. In response to detecting a laser signal in the first set of images, the sensor is caused to operate in a second mode. The laser signal includes an embedded frequency signal component and repeats at a periodic rate. While the sensor is operating in the second mode, the sensor generates a second set of images, which capture an entire period of the laser signal. From the second set of images, the embedded frequency signal component is determined. A decoding operation is performed using the embedded frequency signal component.

Laser pulse decoding using a sensor

A method for operating a depth imaging system is presented. The method comprises illuminating an environment using an optical source configured to output modulated structured light comprising a pattern of dots. Reflected illumination is received at an optical sensor comprising a 2D pixel grid. Received reflected illumination from the 2D pixel grid is downsampled to a plurality of 1D dot trail vectors. For each dot trail vector, a depth is indicated based on a time-of-flight depth measurement and a triangulation factor.

Depth imaging system with downsampled reflected illumination

One example provides a method of operating a time-of-flight camera system comprising an illumination source and an image sensor. The method comprises operating the illumination source and the image sensor to control a plurality of integration cycles and a plurality of readout cycles. In each integration cycle, the method comprises performing a plurality of pulse width modulated (PWM) illumination cycles where each PWM illumination cycle is separated from one or more adjacent PWM illumination cycles by a non-illumination cycle. For each PWM illumination cycle, the method comprises directing photocharge to in-pixel memory for each pixel that is performing image integration and for each non-illumination cycle conducting photocharge away from the in-pixel memory for each pixel that is performing image integration. The readout cycle comprises, for each pixel that performed image integration, reading a charge stored in the in-pixel memory after the integration cycle.

Pulse width modulated integration for time of flight imaging

Examples are disclosed herein that relate to a time-of-flight camera that performs phase unwrapping in an efficient manner. In one example, a time-of-flight camera includes a light emitter, a sensor array, and a controller. The controller is configured to select a frequency mode from a plurality of frequency modes, each frequency mode including two or more different frequencies, and at least two different frequency modes of the plurality of frequency modes having a common frequency shared between the at least two frequency modes, control the light emitter to illuminate a scene with modulated light of the two or more different frequencies of the frequency mode selected, control the sensor array to receive the modulated light reflected from objects within the scene, and process the modulated light received to determine unwrapped phases for the frequency mode selected based on the two or more different frequencies of the frequency mode selected.

Multiple-mode frequency sharing for time-of-flight camera

The technology described herein recalibrates a structured light sensor in the field using time-of-flight sensor data. Structured light sensors are sensitive to mechanical changes that result in decreased accuracy. A structured light system calculates the range to an object by comparing a reference image to the actual image of the scene. The reference image is what the projected light pattern would look like on a flat object at a known distance. When the projected image changes, the reference image no longer matches the projected pattern. The calibration technology described herein captures a new reference image based on the current sensor characteristics using a time-of-flight capable sensor as the structured light imaging sensor.

Field calibration of a structured light range-sensor

One example provides a time-of-flight depth imaging system configured to modulate light emitted from a light source to illuminate an environment with modulated light, and for each of one or more modulation frequencies, integrate an image at each phase step of a plurality of phase steps, and sense a temperature of the light source and/or image sensor via one or more temperature sensors to acquire a measured temperature. The instructions are further executable to, and for each pixel of one or more pixels of the image sensor, determine a complex phasor based upon the measured temperature using a linear inverse function for each modulation frequency, determine a phase shift between the light emitted from the light source and light from the light source reflected back by the environment based on the complex phasor, and output a depth value for the pixel based upon the phase shift.

Time-of-flight measurements using linear inverse function

The technology described herein is a depth camera that uses structured light and modulated light to produce a depth image. The modulated and structured light allows a time-of-flight depth to be calculated for each unit (e.g., a single dot) of the reflected structured light image captured by the depth camera's image sensor. The time-of-flight depth can be used to accurately identify each unit of the reflected structured light image. Once each unit of the reflected structured light image is identified, a structured light triangulation algorithm can be used to calculate a depth for each unit of the reflected structured light image. The present technology also allows a confidence score to be generated for the assigned depths within the depth image by comparing the structured light depth for each unit to the TOF depth for each unit.

Time-of-flight augmented structured light range-sensor

The time-of-flight system disclosed herein includes a frequency unwrapping module configured to generate an input phase vector with M phases corresponding to M sampled signals from an object, determine an M&#x2212;1 dimensional vector of transformed phase values by applying a transformation matrix (T) to the input phase vector, determine an M&#x2212;1 dimensional vector of rounded transformed phase values by rounding the transformed phase values to a nearest integer, and determine a one dimensional lookup table (LUT) index value by transforming the M&#x2212;1 dimensional rounded transformed phase values. The index value is input into the one dimensional LUT to determine a range of the object.

Multi-frequency unwrapping

A method for facilitating removal of multipath signal interference from light data can comprise illuminating, with an illumination unit, a target with a light source. The illumination unit can be configured to project a high spatial-frequency pattern onto the target in such a way as to redistribute spectral energy to higher frequencies. The method can also comprise acquiring, with a sensor unit, reflected light data reflected from the target. The reflected light data can comprise an array of spatial domain information received from light reflected by the target. Further, the method can comprise processing, with the one or more processors, the reflected light data. The processing applies a high-pass filter within the spatial domain to the reflected light data.

Multipath signal removal in time-of-flight camera apparatus

A method for facilitating removal of specular reflection noise from light data can include illuminating, using an illumination unit, a target with a light source. The illumination unit is configured to project light with a spatial light pattern onto the target. The method can also include acquiring, with a sensor unit, light data that is reflected from the target. The light data may comprise a directly reflected spatial light pattern and a specular reflected spatial light pattern. The directly reflected spatial light pattern and the specular reflected spatial light pattern comprise at least one spatial distinction that distinguishes the directly reflected spatial light pattern from the specular reflected spatial light pattern. The method can further comprise processing the light data to distinguish the directly reflected spatial light pattern from the specular reflected spatial light pattern based upon the at least one spatial distinction.

Specular reflection removal in time-of-flight camera apparatus

Data compression is described herein. The encoder transmits a coded word having replacement bits, as well as a code that defines the starting location of the replacement bits in a data sample. The replacement bits may be actual bits from a selected location in the new data sample. The selected location of the replacement bits can vary from data sample to data sample. The encoder may select the location based on the most significant bit that has changed. Thus, reconstructed data will be bit-accurate from the replaced bits all the way to the highest-order bit. A limited number of key values can be transmitted losslessly. Moreover, the data compression does not need forward error correction (FEC), which is a necessary part of many lossy delta encoding schemes. Furthermore, the encoding and decoding can be done very efficiently in terms of hardware and/or software.

Growing community of inventors

Mountain View, CA, United States of America

John Peter Godbaz