Microsoft Technology Licensing, LLC

Gm Cruise Holdings LLC

Zhanping Xu

Travis Perry

Vei-Han Chan

Cyrus Soli Bamji

Robert Vets

Sunil Prakash Acharya

Andrew Dean Payne

Glenn D Sweeney

Kartheek Chandu

Dawson Yee

Barry Thompson

Matthew D Morris

Algird Michael Gudaitis

John Peter Godbaz

Swati Mehta

Abdelrehim Ahmed

Tamer Elkhatib

Gage Krieger Hills

Brock McCray Roland

Mike Fenton

Larry Prather

Aspects of the subject technology relate to systems, methods, and computer-readable media for identifying roll, pitch, and azimuth angles for an object based on surface normal vectors. Image data tracking a surface as an object moves relative to the surface can be accessed. A reference normal vector of the surface can be identified. A dynamic normal vector of the surface associated with the object moving over the surface can be identified from the image data. One or a combination of roll, pitch, and azimuth of the object with respect to the surface can be determined based on the reference normal vector and the dynamic normal vector.

Identifying stability of an object based on surface normal vectors

Aspects of the subject technology relate to systems, methods, and computer-readable media for performing ground segmentation through formation of a single super voxel. Image data tracking a surface as an object moves relative to the surface is accessed. Samples of the image data is grouped into different subsets to form a plurality of subsets of the image data. A normal vector to a corresponding plane defined by each subset of the plurality of subsets of the image data are identified. The samples of the image data are re-grouped into a single subset of the image data based on the corresponding normal vector for each subset of the plurality of subsets of the image data. A surface plane representation of the surface is identified based on the single subset of the image data.

Ground segmentation through super voxel

Adaptive phase unwrapping for a time-of-flight camera. A scene is illuminated with modulation light having two or more frequencies that do not have a common integer denominator. The modulation light that is reflected off objects within the scene is received at a sensor array. The received modulation light is then processed and weighted in the complex domain to determine unwrapped phases for each of the two or more frequencies of modulation light.

Unwrapped phases for time-of-flight modulation light

A three-dimensional camera having a micro lens (ML) array configured with variable ML height and variable ML shift is described herein. The ML array includes micro lens that are configured to direct backscattered light that is transmitted through an image lens into corresponding pixels. Heights of individual micro lenses within the ML array vary according to image height. For example, the height of micro lenses at the center of the ML array, near the axis of the image lens, may be relatively larger than the height of other micro lenses toward the perimeter of the ML array. Furthermore, the shift of individual micro lenses with respect to corresponding pixels may also vary according to the image height. For example, the shift of micro lenses at the center of the ML array may be relatively smaller than the shift of the other micro lenses toward the perimeter of the ML array.

Micro lens sensor having micro lens heights that vary based on image height

Examples are disclosed herein that relate to a time-of-flight camera that performs phase unwrapping in an efficient manner. In one example, a time-of-flight camera includes a light emitter, a sensor array, and a controller. The controller is configured to select a frequency mode from a plurality of frequency modes, each frequency mode including two or more different frequencies, and at least two different frequency modes of the plurality of frequency modes having a common frequency shared between the at least two frequency modes, control the light emitter to illuminate a scene with modulated light of the two or more different frequencies of the frequency mode selected, control the sensor array to receive the modulated light reflected from objects within the scene, and process the modulated light received to determine unwrapped phases for the frequency mode selected based on the two or more different frequencies of the frequency mode selected.

Multiple-mode frequency sharing for time-of-flight camera

A three-dimensional time-of-flight (TOF) RGB-IR image sensor is provided, including a signal generator configured to generate a modulated electrical signal. The three-dimensional TOF RGB-IR image sensor may further include a light-emitting diode (LED) configured to receive the modulated electrical signal and emit modulated light. The three-dimensional TOF RGB-IR image sensor may further include a TOF sensor integrated circuit configured to receive light at the light-receiving surface and generate a photoelectrical signal based on the received light. The received light may include ambient light and reflected modulated light. The three-dimensional TOF RGB-IR image sensor may further include a filter array located on the light-receiving surface of the TOF sensor integrated circuit. The filter array may include a plurality of pixels, each pixel including an infrared-transmitting bandpass filter and one or more visible-light-transmitting bandpass filters located adjacent to the infrared-transmitting bandpass filter.

Time-of-flight RGB-IR image sensor

A three-dimensional time-of-flight (3D TOF) camera having a micro lens (ML) array configured with variable ML height and variable ML shift. The ML array includes micro lens that are configured to direct backscattered light that is transmitted through an image lens into corresponding pixels. Heights of individual micro lenses within the ML array vary according to image height. For example, the height of micro lenses at the center of the ML array, near the axis of the image lens, may be relatively larger than the height of other micro lenses toward the perimeter of the ML array. Furthermore, the shift of individual micro lenses with respect to corresponding pixels may also vary according to the image height. For example, the shift of micro lenses at the center of the ML array may be relatively smaller than the shift of the other micro lenses toward the perimeter of the ML array.

Micro lens time-of-flight sensor having micro lens heights that vary based on image height

A range camera operable to provide a range image of a scene and a picture of the scene, based on non-visible light and visible light registered for features in the scene on a same photosensor.

Time of flight and picture camera

A continuous wave time of flight (CW-TOF) camera that modulates sensitivity of its photosensor during an exposure period at a frequency of modulation of structured light that the camera transmits to illuminate a scene that it images but phase shifted relative to phase of the transmitted light by a sampling phase offset modified by a sequence of N perturbation phase shifts &#x3bb;=2&#x3c0;(n&#x2212;1)/N, (1&#x2264;n&#x2264;N), and modifies the structured light in synchrony with the sequence of perturbation phase shifts to reduce error due to multipath interference in determining distances to features in the scene.

Time of flight camera

A method of determining distances to features in a scene comprising transmitting structured light modulated at a modulation frequency to illuminate the scene with a structured illumination pattern and for each combination of a sampling phase offset &#x3c8;and a perturbation phase shift &#x3bb;, modulating sensitivity of a photo sensor at the frequency of modulation of the transmitted light, but phase shifted relative to phase of the transmitted light by a phase &#x3b8;=(&#x3c8;+&#x3bb;), and for each value of &#x3b8;modulo 360&#xb0;, registering light reflected by features in the scene during a different exposure and using the registered light to provide a range image of the scene.

A method for determining whether a distance that a CW-TOF range camera provides for a scene is degraded by multipath interference (MPI) comprising operating the camera to determine a propagation phase delay and a phase delay coefficient for each of a plurality of modulation frequencies of light that illuminates the scene and using the phase delay coefficient and/or the phase delay to determine whether a distance provided by the camera is compromised by MPI.

A continuous wave time of flight (CW-TOF) camera operable to determine distances to features in a scene that are corrected for light that the camera transmits that is back scattered to a photosensor in the camera by structural features of the camera.

Imaging sensors that detect infrared and visible light are provided herein. In one example, an imaging sensor is presented that includes a semiconductor substrate comprising an array of pixel structures for concurrently sensing infrared light and visible light. Each of the pixel structures include a first pixel element configured to detect the infrared light and a second pixel element configured to detect the visible light. Each of the pixel structures further include a shared output circuit that couples the first pixel element and the second pixel element such that a first output state presents a first signal corresponding to detected infrared light of the first pixel element and a second output state presents a second signal corresponding to detected visible light of the second pixel element.

Imaging sensor with shared pixel readout circuitry

A CW-TOF camera that uses a piecewise constant or linear discretized indicator function of first and second modulation frequencies of light that the camera transmits to illuminate a scene and a round trip time tfor light from and back to the camera for features in the scene to disambiguate wrapped phase shifts that the camera acquires for the features.

Growing community of inventors

Sunnyvale, CA, United States of America

Zhanping Xu