Zoox, Inc.

Waymo LLC

Planet Labs, Inc.

Joseph Patrick Warga

Ryan McMichael

Robert Nicholas Moor

Subasingha Shaminda Subasingha

Harrison Thomas Waschura

Silas Kogure Wilkinson

Turhan Karadeniz

Christopher William Labadie

Derek Adams

Daniel Glenn Johnson

Daniel Miller

Anubhav Thakur

Yongzhe Chen

Samuel Holladay

Austin In-Jei Yi

Peter Thomas Mitros

Mehran Ferdowsi

Ralph Hamilton Shepard

Lucian Ion

Denis Nikitin

Holly Ho

Benjamin Haldeman

Clark Pentico

Lee Tsung

Example embodiments relate to foveated imagers for automotive applications. An example embodiment includes a device. The device includes a rotationally symmetric foveated lens. The rotationally symmetric foveated lens is configured to receive light from an environment. The rotationally symmetric foveated lens is also configured to produce an image at an image plane based on the received light. The device also includes an image sensor having an associated image sensor resolution. The image sensor is positioned at the image plane and configured to capture an image having an associated field of view of the environment. Based on a distortion profile of the rotationally symmetric foveated lens and the image sensor resolution, the captured image exhibits a first angular optical resolution in a central region of the field of view and a second angular optical resolution in a peripheral region of the field of view.

Foveated imager for automotive applications

Techniques for determining a distance to an object in an environment based on sensor data are discussed herein. The sensor data can be captured by a sensor, and the sensor data can be based on laser pulses emitted from emitters being differently spaced apart from the sensor. The sensor data can be utilized to determine distance data associated with an object. A difference between the distance data can be used to determine whether the distance data is associated with a retroreflector. If the difference in distance data is above a threshold, a confidence level of sensor data can be adjusted or the sensor data can be discarded or omitted from subsequent processing.

Glare detection in sensor data

Sensors, including time-of-flight sensors, may be used to detect objects in an environment. In an example, a vehicle may include a time-of-flight sensor that images objects around the vehicle, e.g., so the vehicle can navigate relative to the objects. The sensor may generate first image data at a first configuration and second image data at a second configuration. An estimated depth of an object may be determined from the first image data, and an actual depth of the object may be determined from the second image data, based on the estimated depth. In examples, the first and second configurations have different modulation frequencies such that a nominal maximum depth in the first configuration is greater than the nominal maximum depth in the second configuration.

Determining depth using multiple modulation frequencies

Sensors, including time-of-flight sensors, may be used to detect objects in an environment. In an example, a vehicle may include a time-of-flight sensor that images objects around the vehicle, e.g., so the vehicle can navigate relative to the objects. Sensor data generated by the time-of-flight sensor can be impacted by glare. In some examples, an emitter system for illuminating a field of view can be dynamically altered to provide different illumination intensities at different regions in the field of view. For instance, locations at which a highly-reflective object or other object that is likely to cause glare may be illuminated at a lower illumination intensity, e.g., to reduce the likelihood of pixel saturation in measured data associated with the object.

Dynamic sensor illumination

Sensors, including time-of-flight sensors, may be used to detect objects in an environment. In an example, a vehicle may include a time-of-flight sensor that images objects around the vehicle, e.g., so the vehicle can navigate relative to the objects. Sensor data generated by the time-of-flight sensor can include returns associated with highly reflective objects that cause glare. In some examples, a depth of a sensed surface is determined from the sensor data and additional pixels at the same depth are identified. The subset of pixels at the depth are filtered by comparing a measured intensity value to a threshold intensity value for the depth. Other threshold intensity values can be applied to subsets of pixels at different depths.

Depth dependent pixel filtering

A sensor pod system includes one or more sensor pods with a plurality of sensors configured to collect data from an environment. A sensor pod may have an effective field of view created by individual sensors with overlapping fields of view. The sensor pod system may include sensors of different types and modalities. Sensor pods of the sensor pod system may be modularly installed on a vehicle, for example, an autonomous vehicle and collect and provide data of the environment during operation of the vehicle.

Sensor pod assembly

A lens assembly for systems such as imaging devices. The lens assembly may include at least five lens elements, where one or more of the lens elements includes positive optical power and one or more of the lenses includes negative optical power. The lens assembly may further include an aperture stop located between the lens elements. In some instances, the lens assembly provides a horizontal field of view that is at least 80 degrees while still including a total track length that is less than or equal to 50 millimeters and a diameter that is less than or equal to 13 millimeters. Additionally, in some instances, the lens assembly may cause all rays which impinge the sensor to be less than or equal to approximately 10 degrees.

Lens assembly for sensors

Sensors, including time-of-flight sensors, may be used to detect objects in an environment. In an example, a vehicle may include a time-of-flight sensor that images objects around the vehicle, e.g., so the vehicle can navigate relative to the objects. The sensor may generate first image data at a first configuration and second image data at a second configuration. A disambiguated depth of a surface may be determined from the first image data and the second image data. If the disambiguated depth is greater than a nominal maximum depth of the sensor in the first configuration, an intensity of the surface may be determined from the first image data. If the intensity meets or exceeds a threshold intensity, the surface may be determined to be beyond the nominal maximum depth. If the intensity is less than the threshold intensity, an actual depth of the surface may be determined form the second image data as a distance less than the nominal maximum depth.

Determining pixels beyond nominal maximum sensor depth

A wind tunnel test may be performed on a vehicle to determine accumulation of substances (e.g., water) on sensors of the vehicle. Control surfaces may be created for the sensors based images representing the accumulations, where the control surfaces represent the include obstructions located where accumulations were detected during the test. The vehicle may then navigate around an environment using the control surfaces in order to determine a drivability of the vehicle. Also, a simulation may be performed, where the simulation outputs images representing simulated accumulations on the sensors. The outputs from the simulation may be compared to the results from the test in order to determine how accurately the simulation represents the test, determine domains in which the vehicle may safely operate, and/or improve the simulation.

Vehicle sensor degradation testing

A sensor pod system includes one or more sensor pods with a plurality of sensors configured to collect data from an environment. The sensor pod system may include a cleaning system to clean sensing surfaces of sensor pods during operation. The sensor pod system may include sensors of different types and modalities. Sensor pods of the sensor pod system may be modularly installed on a vehicle, for example, an autonomous vehicle and collect and provide data of the environment during operation of the vehicle.

Sensor pod cleaning system

Sensors, including time-of-flight sensors, may be used to detect objects in an environment. In an example, a vehicle may include a time-of-flight sensor that images objects around the vehicle, e.g., so the vehicle can navigate relative to the objects. Sensor data generated by the time-of-flight sensor can include saturated pixels, e.g., due to over-exposure, sensing highly-reflective objects, and/or excessive ambient light. In some examples, parameters associated with power of a time-of-flight sensor can be altered based on characteristics of the saturated pixels, as well as information about non-saturated pixels neighboring the saturated pixels. For example, the neighboring pixels may provide information about whether saturation is due to ambient light, e.g., sunlight, or due to emitted light from the sensor.

Active power control using neighboring pixels

A sensor pod system includes one or more sensor pods with a plurality of sensors configured to collect data from an environment. A sensor pod may have an effective field of view created by individual sensors with overlapping fields of view. The sensor pod system may include sensors of different types and modalities. Sensor pods of the sensor pod system may be modularly disposed on a vehicle, for example, an autonomous vehicle to collect and provide data of the environment during operation of the vehicle. The sensor pods may be disposed at elevated locations around the vehicle to reduce obstacles within the sensor pods fields of view.

Sensor pod coverage and placement on vehicle

Various lens designs for use in Time of Flight (ToF) sensor systems are discussed. Improvements to a ToF sensor system may be realized by, for example, incorporating a lens having particular features, such as a relatively short track length, fast lens speed (e.g., low f-number), low telecentricity, relatively flat field illumination, and fairly low cost. In some examples, such a lens of a ToF sensor system may be a lens assembly having a fixed focal length and that avoids use of lenses having aspheric surfaces so as to achieve relatively low cost.

Time of flight sensor system and lens

A sensor pod system includes one or more sensor pods with a plurality of sensors configured to collect data from an environment. A sensor pod may have an effective field of view created by individual sensors with overlapping fields of view. The sensor pod system may include sensors of different types and modalities. Sensor pods of the sensor pod system may be modularly installed on a vehicle, for example, an autonomous vehicle and collect and provide data of the environment during operation of the vehicle. A combination of techniques may be used to calibrate a sensor pod prior to installation in the vehicle and may be used to determine if the sensor pod is compatible with other sensor pods and may also be used to calibrate other sensors when integrating the sensor pod into the vehicle.

Sensor pod calibration

A sensor pod system includes one or more sensor pods with a plurality of sensors configured to collect data from an environment. A sensor pod may include a housing and extend from a portion of a body of a vehicle. The sensor pod housing may have energy absorbing structures configured to absorb and dissipate energy during an impact in order to protect a pedestrian. The sensor pod may have deformable portions of the housing configured to absorb and dissipate energy during the impact. The sensor pod may have deformable fasteners coupling a sensor to the sensor pod configured to deform to absorb and dissipate energy during the impact.

Growing community of inventors

San Francisco, CA, United States of America

Joseph Patrick Warga