Wisconsin Alumni Research Foundation

Sony Corporation

Snap Inc.

Columbia University

Mitsubishi Electric Research Laboratories, Inc.

Mohit Gupta

Shree Kumar Nayar

Andreas Udo Velten

Atul Nishikant Ingle

Tomoo Mitsunaga

Felipe Gutierrez Barragan

Jongho Lee

Sizhuo Ma

Yasunobu Hitomi

Mikhail Fridberg

Daisuke Iso

Jinwei Gu

Brandon M Smith

Anant Gupta

Eric Breitbach

Yuhao Liu

Vishwanath Saragadam Raja Venkata

Amit Kumar Agrawal

Kensei Jo

Ashok Veeraraghavan

Vishal R Agarwal

Varun Sundar

Matthew Dutson

Shantanu Gupta

Andreas Velten

Pratham H Desai

In accordance with some embodiments, systems, methods, and media for high dynamic range imaging using single-photon and conventional image sensor data are provided. In some embodiments, the system comprises: first detectors configured to detect a level of photons proportional to incident photon flux; second detectors configured to detect arrival of individual photons; a processor programmed to: receive, from the first detectors, first values indicative of photon flux from a scene with a first resolution; receive, from the second detectors, second values indicative of photon flux from the scene with a lower resolution; provide a first encoder of a trained machine learning model first flux values based on the first values, provide the second encoder of the model second flux values; receive, as output, values indicative of photon flux from the scene; and generate a high dynamic range image based on the third plurality of values.

Systems, methods, and media for high dynamic range imaging using single-photon and conventional image sensor data

In accordance with some embodiments, systems, methods, and media for estimating depths in a scene using single-photon detectors and binary structured light patterns are provided. In some embodiments, a system comprises: a light source; an image sensor comprising an array including single-photon detectors; a processor programmed to: cause the light source to emit a sequence of n binary light patterns toward the scene, each comprising at least L columns, each of the patterns has a minimum stripe width of 8 columns; and n>log(L); cause, for each of the n binary light patterns, the image sensor to capture a binary image of the scene, the binary image comprises a binary value for each of single-photon detectors such that each of the single-photon detectors generates a set of binary values; and estimate, for each single-photon detector, a depth value based on the set of binary values.

Systems, methods, and media for estimating depths in a scene using single-photon detectors and binary structured light patterns

In accordance with some embodiments, systems, methods, and media for Eulerian single-photon computer vision are provided. In some embodiments, the system comprises: an image sensor comprising detectors configured to detect arrival of individual photons, and arranged in an array; a processor programmed to: cause the image sensor to generate a sequence of images representing a scene, each of the images comprising a plurality of pixels; perform, for each of a plurality of three dimensional filters, a convolution between the three dimensional filter and a plurality of frames, wherein each of the plurality of frames is based on one or more of the images of the sequence of images; generate, for each of the plurality of frames, a plurality of filter bank responses each corresponding to a three dimensional filter of the plurality of three dimensional filters; and perform a computer vision process based on the plurality of filter responses.

Systems, methods, and media for eulerian single-photon computer vision

In accordance with some embodiments, systems, methods, and media for generating digital images using low bit depth image sensor data are provided. In some embodiments, the system comprises: an image sensor; a processor programmed to: receive, from the image sensor, a series of low bit depth frames; provide low bit depth image information to a trained machine learning model comprising: a 3D convolutional layer; a 2D convolutional LSTM layer; a concatenation layer configured to generate a tensor that includes an output of the 2D convolutional LSTM layer and the low bit depth image information; and a 2D convolutional layer configured to generate an output based on the tensor; and generate a high bit depth image of a scene based on an output of the two-dimensional convolutional layer.

Systems, methods, and media for generating digital images using low bit depth image sensor data

In accordance with some embodiments, systems, methods and media for single photon depth imaging with improved precision in ambient light conditions are provided. In some embodiments, the system comprises: a light source; a single photon detector; an attenuation element configured to provide a variable intensity attenuation factor; and a processor programmed to: (a)-determine an ambient light intensity associated with a scene point; (b)-select an attenuation factor based on the ambient light intensity; (c)-estimate a depth of the scene point based on a multiplicity of photon arrival times determined using the detector during a period of time during which light incident on the detector is attenuated by the selected attenuation factor and during which the light source is configured to periodically emit a pulse of light toward the scene point; (d)-repeat (a)-(c) for each of a multiplicity of scene points.

Systems, methods, and media for single photon depth imaging with improved precision in ambient light

In accordance with some embodiments, systems, methods, and media for improving signal-to-noise ratio in single-photon data are provided. In some embodiments, a system comprises: a processor programmed to: generate, for each of a plurality of pixel locations, initial photon flux estimates based on a first set of photon transients including photon transients associated with the pixel location and neighboring pixel locations, each of the photon transients comprises a histogram of photon counts; identify, for a scene patch associated with each of the plurality of pixel locations, one or more similar scene patches using intensity information for each of the plurality of pixel locations; and generate, for each of the plurality of pixel locations, final photon flux estimates based on a second set of photon transients including photon transients associated with the scene patch and each of the one or more similar scene patches.

Systems, methods, and media for improving signal-to-noise ratio in single-photon data

A client device, such as a mobile phone or a mobile phone accessory (e.g., phone case), is provided that receives and transmits data (e.g., a social media code) via light wave communication. The light wave communication may comprise structured light (e.g., projected light patterns). The client device may include a lightbox comprised of LEDs located on a back face of the client device.

Low power light wave communication for mobile and wearable devices

In accordance with some embodiments, systems, methods and media for high dynamic range quanta burst imaging are provided. In some embodiments, the system comprises: an image sensor comprising single photon detectors in an array; a processor programmed to: generate a sequence of binary images representing a scene; divide the sequence of binary images into blocks; generate block-sum images from the blocks; determine alignments between the block-sum images and a reference block-sum image; warp the sequence of binary images based on the alignments; generate warped block-sum images using warped binary images; merge the warped block-sum images; display a final image of the scene based on the merged warped block-sum images.

Systems, methods, and media for high dynamic range quanta burst imaging

In accordance with some embodiments, systems, methods and media for stochastic exposure coding for continuous time-of-flight imaging are provided. In some embodiments, a method for estimating the depth of a scene is provided, comprising: stochastically selecting active slots based on a probability p; causing, during active slots, a light source to emit light modulated by a first modulation function toward a scene; causing, during active slots, an image sensor to generate a first, second, and third value based on received light from a portion of the scene and a first, second, and third demodulation function, respectively; inhibiting the light source during inactive slots; determining, for each of the active slots, depth estimates for the portion of the scene based on the first, second, and third value; and determining a depth estimate for the portion of the scene based on the depth estimates for the active slots.

Systems, methods, and media for stochastic exposure coding that mitigates multi-camera interference in continuous wave time-of-flight imaging

An augmented reality system having a light source and a camera. The light source projects a pattern of light onto a scene, the pattern being periodic. The camera captures an image of the scene including the projected pattern. A projector pixel of the projected pattern corresponding to an image pixel of the captured image is determined. A disparity of each correspondence is determined, the disparity being an amount that corresponding pixels are displaced between the projected pattern and the captured image. A three-dimensional computer model of the scene is generated based on the disparity. A virtual object in the scene is rendered based on the three-dimensional computer model.

Augmented reality system using structured light

In accordance with some embodiments, systems, methods, and media for asynchronous single photon depth imaging with improved precision in ambient light conditions are provided. In some embodiments, the system comprises: a light source; a detector configured to detect arrival of individual photons, and enter a dead time after a detection; a processor programmed to: cause the light source to emit pulses toward a scene point at the beginning of light source cycles each corresponding to B time bins; cause the detector to enter an acquisition window at a first time bin position; cause the detector to enter another acquisition window at a shifted time bin position; record photon arrival times; associate each photon arrival time with a time bin; and estimate a depth of the scene point based on a number of photon detection events at each time bin, and a denominator corresponding to each time bin.

Systems, methods, and media for asynchronous single photon depth imaging with improved precision in ambient light

In accordance with some embodiments, systems, methods and media for directly recovering planar surfaces in a scene using structured light are provided. In some embodiments, a system comprises: a light source; an image sensor; a processor programmed to: cause the light source to emit a pattern comprising a pattern feature with two line segments that intersect on an epipolar line; cause the image sensor to capture an image including the pattern; identify an image feature in the image, the image feature comprising two intersecting line segments that intersect at a point in the image that corresponds to the first epipolar line; estimate a plane hypothesis associated with the pattern feature based on properties of the pattern feature and properties of the image feature, the plane hypothesis associated with a set of parameters characterizing a plane; and identify a planar surface in the scene based on the plane hypothesis.

Growing community of inventors

Madison, WI, United States of America

Mohit Gupta