Wisconsin Alumni Research Foundation

Massachusetts Institute of Technology

Columbia University

Andreas Udo Velten

Ramesh Raskar

Mohit Gupta

Atul Nishikant Ingle

Shree Kumar Nayar

Felipe Gutierrez Barragan

Eric Breitbach

Trevor Seets

Adam Uselmann

Thomas Rockwell Mackie

Kevin Eliceiri

Rohit Pandharkar

Anant Gupta

Yuhao Liu

Christie Lin

Surendra Prajapati

Wei Lin

Yizou Lu

Marco La Manna

Real time, low-light images, for example, obtained from the fluorescent marker for identifying tumors during surgery, are combined to improve the signal-to-noise ratio using a motion signal derived from corresponding high-light images, for example, taken with a second camera at interleaved intervals of higher illumination.

Low-light video system

In accordance with some embodiments, systems, methods and media for single photon depth imaging with improved precision in ambient light conditions are provided. In some embodiments, the system comprises: a light source; a single photon detector; an attenuation element configured to provide a variable intensity attenuation factor; and a processor programmed to: (a)-determine an ambient light intensity associated with a scene point; (b)-select an attenuation factor based on the ambient light intensity; (c)-estimate a depth of the scene point based on a multiplicity of photon arrival times determined using the detector during a period of time during which light incident on the detector is attenuated by the selected attenuation factor and during which the light source is configured to periodically emit a pulse of light toward the scene point; (d)-repeat (a)-(c) for each of a multiplicity of scene points.

Systems, methods, and media for single photon depth imaging with improved precision in ambient light

In accordance with some embodiments, systems, methods, and media for high dynamic range imaging using single-photon and conventional image sensor data are provided. In some embodiments, the system comprises: first detectors configured to detect a level of photons proportional to incident photon flux; second detectors configured to detect arrival of individual photons; a processor programmed to: receive, from the first detectors, first values indicative of photon flux from a scene with a first resolution; receive, from the second detectors, second values indicative of photon flux from the scene with a lower resolution; provide a first encoder of a trained machine learning model first flux values based on the first values, provide the second encoder of the model second flux values; receive, as output, values indicative of photon flux from the scene; and generate a high dynamic range image based on the third plurality of values.

Systems, methods, and media for high dynamic range imaging using single-photon and conventional image sensor data

In accordance with some embodiments, systems, methods, and media for motion adaptive imaging using single-photon image sensor data are provided. In some embodiments, the system comprises: an image sensor comprising single-photon detectors in an array; a processor programmed to: receive a sequence of photon frames, each comprising pixels having a value indicative of whether a photon was received during a frame period, each of the pixels corresponds to a pixel location; identify, for each of the pixel locations, changepoints, each indicative of a change in scene brightness; identify a photon frame in the sequence at which at least a threshold change in brightness has occurred based on the changepoints associated with each of the plurality of pixel locations; and generate a series of changepoint frames, wherein each changepoint frame is based on estimated brightness associated with each pixel location at a point in the sequence of photon frames.

Systems, methods, and media for motion adaptive imaging using single-photon image sensor data

An improved non-line-of-sight camera provides for real-time evaluation of a relay wall with respect to illuminated points and sensing areas for higher accuracy and practical field use. Gated sensing allows improved recovery of faint photon signals and higher resolution. The system allows an operator to a find virtual camera from looking around multiple corners.

Non-line-of-sight imaging system for distant measurement

In accordance with some embodiments, systems, methods and media for encoding and decoding signals used in time-of-flight imaging are provided. In some embodiments, a method for estimating the depth of a scene is provided, comprising: causing a light source to emit modulated light toward the scene based on a modulation function; causing the image sensor to generate a first value based on the modulated light and a first demodulation function of K modulation functions; causing the image sensor to generate a second value; causing the image sensor to generate a third value; and determining a depth estimate for the portion of the scene based on the first value, the second value, the third value, and three correlation functions each including at least one half of a trapezoid wave.

Systems, methods, and media for encoding and decoding signals used in time of flight imaging

In accordance with some embodiments, systems, methods and media for encoding and decoding signals used in time-of-flight imaging are provided. In some embodiments, a method for estimating the depth of a scene is provided, comprising: causing a light source to emit modulated light toward the scene based on a modulation function; causing the image sensor to generate a first value based on the modulated light and a first demodulation function of K modulation functions, including at least one trapezoid wave; causing the image sensor to generate a second value; causing the image sensor to generate a third value; and determining a depth estimate for the portion of the scene based on the first value, the second value, and the third value.

In accordance with some embodiments, systems, methods and media for high dynamic range imaging using dead-time-limited single photon detectors are provided. In some embodiments, a system for high dynamic range imaging is provided, comprising: an image sensor comprising: a pixels comprising: a single photon detector having dead time &#x3c4;; and a counter coupled to an output of the single photon detector, wherein the counter is configured to increment in response to a signal indicative of detection of a photon output by the single photon detector; and a processor that is programmed to: read out a value stored by the counter after an exposure time has elapsed; and calculate an intensity for the pixel based on the value and the dead time &#x3c4;.

Systems, methods, and media for high dynamic range imaging using dead-time-limited single photon detectors

Fluorescent markers used to identify a tissue may be imaged in a bright environment by synchronizing the imaging process with rapidly switched ambient lighting so that imaging occurs in phase with a switching off of the ambient lighting. In this way, valuable fluorescent imaging may be performed in an environment that appears to be brightly illuminated, for example in the area of a surgical suite.

Tissue fluorescence monitor with ambient light rejection

An active imaging system, which includes a light source and light sensor, generates structured illumination. The light sensor captures transient light response data regarding reflections of light emitted by the light source. The transient light response data is wavelength-resolved. One or more processors process the transient light response data and data regarding the structured illumination to calculate a reflectance spectra map of an occluded surface. The processors also compute a 3D geometry of the occluded surface.

Methods and apparatus for imaging of occluded objects

In exemplary implementations of this invention, a time of flight camera (ToF camera) can estimate the location, motion and size of a hidden moving object, even though (a) the hidden object cannot be seen directly (or through mirrors) from the vantage point of the ToF camera (including the camera's illumination source and sensor), and (b) the object is in a visually cluttered environment. The hidden object is a NLOS (non-line-of-sight) object. The time of flight camera comprises a streak camera and a laser. In these exemplary implementations, the motion and absolute locations of NLOS moving objects in cluttered environments can be estimated through tertiary reflections of pulsed illumination, using relative time differences of arrival at an array of receivers. Also, the size of NLOS moving objects can be estimated by backprojecting extremas of NLOS moving object time responses.

Methods and apparatus for estimation of motion and size of non-line-of-sight objects

In exemplary implementations of this invention, a 3D range camera 'looks around a corner' to image a hidden object, using light that has bounced (reflected) off of a diffuse reflector. The camera can recover the 3D structure of the hidden object.

Methods and apparatus for imaging of occluded objects from scattered light

In exemplary implementations of this invention, a set of two scanning mirrors scans the one dimensional field of view of a streak camera across a scene. The mirrors are continuously moving while the camera takes streak images. Alternately, the mirrors may only between image captures. An illumination source or other captured event is synchronized with the camera so that for every streak image the scene looks different. The scanning assures that different parts of the scene are captured.

Growing community of inventors

Madison, WI, United States of America

Andreas Udo Velten