Massachusetts Institute of Technology

Ayush Bhandari

Ramesh Raskar

Achuta Kadambi

Christopher Barsi

Barmak Heshmat Dehkordi

Vage Taamazyan

Adrian Andrew Dorrington

Boxin Shi

Lee Vincent Streeter

Tomohiro Maeda

James Schiel

Felix Krahmer

Hang Zhao

Refael Whyte

A time-of-flight imaging system may output light with a modulation frequency in the gigahertz band, to illuminate a range target. This high-frequency illumination may enable extremely precise&#x2014;e.g., micron-scale&#x2014;depth measurements. The system may modulate reflected light from the range target, to create a beat tone that has a frequency in the hertz band. In some cases, the modulated light in the gigahertz band is created by a first modulator and the beat tone in the hertz band is created by a second modulator. In some cases, the modulated light in the gigahertz band is created by an upshift cascade of modulators and the beat tone in the hertz band is created by a downshift cascade of modulators. A photodetector may measure the low-frequency beat tone. From this beat tone, phase of the signal and depth of the range target may be extracted.

Methods and apparatus for gigahertz time-of-flight imaging

A self-reset ADC may take a set of temporally equidistant, modulo samples of a bandlimited, analog signal, at a sampling rate that is greater than &#x3c0;e samples per second, where &#x3c0; is Archimedes' constant andis Euler's number. The bandlimited signal may have a bandwidth of 1 Hertz and a maximum frequency of 0.5 Hertz. These conditions of sampling rate, bandwidth and maximum frequency may ensure that an estimated signal may be recovered from the set of modulo samples. This estimated signal may be equal to the bandlimited signal plus a constant. The constant may be equal to an integer multiple of the modulus of the centered modulo operation employed to take the modulo samples.

Methods and apparatus for modulo sampling and recovery

A light source may illuminate a scene with amplitude-modulated light. The scene may include fluorescent material. The amplitude modulation may be periodic, and the frequency of the amplitude modulation may be swept. During the sweep, a time-of-flight sensor may take measurements of light returning from the scene. A computer may calculate, for each pixel in the sensor, a vector of complex numbers. Each complex number in the vector may encode phase and amplitude of light incident at the pixel and may correspond to measurements taken at a given frequency in the sweep. A computer may, based on phase of the complex numbers for a pixel, calculate fluorescence lifetime and scene depth of a scene point that corresponds to the pixel.

Methods and apparatus for fluorescence lifetime imaging with periodically modulated light

A light source may illuminate a scene with pulsed light that is pulsed non-periodically. The scene may include fluorescent material that fluoresces in response to the pulsed light. The pulsed light signal may comprise a maximum length sequence or Gold sequence. A lock-in time-of-flight sensor may take measurements of light returning from the scene. A computer may, for each pixel in the sensor, perform a Discrete Fourier Transform on measurements taken by the pixel, in order to calculate a vector of complex numbers for the pixel. Each complex number in the vector may encode phase and amplitude of incident light at the pixel and may correspond to measurements taken at a given time interval during the pulsed light signal. A computer may, based on phase of the complex numbers for a pixel, calculate fluorescence lifetime and scene depth of a scene point that corresponds to the pixel.

Methods and apparatus for fluorescence lifetime imaging with pulsed light

In some implementations, scene depth is extracted from dual frequency of a cross-correlation signal. A camera may illuminate a scene with amplitude-modulated light, sweeping the modulation frequency. For each modulation frequency in the sweep, each camera pixel may measure a cross-correlation of incident light and of a reference electrical signal. Each pixel may output a vector of cross-correlation measurements acquired by the pixel during a sweep. A computer may perform an FFT on this vector, identify a dual frequency at the second largest peak in the resulting power spectrum, and calculate scene depth as equal to a fraction, where the numerator is the speed of light times this dual frequency and the denominator is four times pi. In some cases, the two signals being cross-correlated have the same phase as each other during each cross-correlation measurement.

Methods and apparatus for time-of-flight imaging

A time-of-flight camera images an object around a corner or through a diffuser. In the case of imaging around a corner, light from a hidden target object reflects off a diffuse surface and travels to the camera. Points on the diffuse surface function as a virtual sensors. In the case of imaging through a diffuser, light from the target object is transmitted through a diffusive media and travels to the camera. Points on a surface of the diffuse media that is visible to the camera function as virtual sensors. In both cases, a computer represents phase and intensity measurements taken by the camera as a system of linear equations and solves a linear inverse problem to (i) recover an image of the target object; or (ii) to compute a 3D position for each point in a set of points on an exterior surface of the target object.

Methods and apparatus for virtual sensor array

In illustrative implementations, a time-of-flight camera robustly measures scene depths, despite multipath interference. The camera emits amplitude modulated light. An FPGA sends at least two electrical signals, the first being to control modulation of radiant power of a light source and the second being a reference signal to control modulation of pixel gain in a light sensor. These signals are identical, except for time delays. These signals comprise binary codes that are m-sequences or other broadband codes. The correlation waveform is not sinusoidal. During measurements, only one fundamental modulation frequency is used. One or more computer processors solve a linear system by deconvolution, in order to recover an environmental function. Sparse deconvolution is used if the scene has only a few objects at a finite depth. Another algorithm, such as Wiener deconvolution, is used is the scene has global illumination or a scattering media.

Methods and apparatus for coded time-of-flight camera

In illustrative implementations of this invention, an imaging system includes multiple light sources that illuminate a scene, and also includes a lock-in time of flight camera. While the scene is illuminated by these light sources, each of the light sources is amplitude-modulated by a different modulation pattern, and a reference signal is applied to the lock-in time-of-flight camera. The modulation patterns and the reference signal are carefully chosen such that the imaging system is able to disentangle, in real time, the respective contributions of the different light sources, and to compute, in real-time, depth of the scene. In some cases, the modulation signals for the light sources are orthogonal to each other and the reference signal is broadband. In some cases, the modulation codes for the light sources and the reference code are optimal codes that are determined by an optimization algorithm.

Methods and apparatus for demultiplexing illumination

In exemplary implementations of this invention, a multi-frequency ToF camera mitigates the effect of multi-path interference (MPI), and can calculate an accurate depth map despite MPI. A light source in the multi-frequency camera emits light in a temporal sequence of different frequencies. For example, the light source can emit a sequence of ten equidistant frequencies f=10 MHz, 20 MHz, 30 MHz, . . . , 100 MHz. At each frequency, a lock-in sensor within the ToF camera captures 4 frames. From these 4 frames, one or more processors compute, for each pixel in the sensor, a single complex number. The processors stack all of such complex quantities (one such complex number per pixel per frequency) and solve for the depth and intensity, using a spectral estimation technique.

Growing community of inventors

Cambridge, MA, United States of America

Ayush Bhandari