Waymo LLC

Semiconductor Components Industries, LLC

Nirav Dharia

Lucian Ion

Vlad Constantin Cardei

Benjamin Frevert

Caner Onal

Xiaoying He

Giulia Guidi

Pierre-Yves Droz

Ralph Hamilton Shepard

Damien Kelly

Ignacio Garcia Dorado

Pawan Gilhotra

Andy Wong

Xiaoli Lu

Jia Feng

Evan Cull

A method includes obtaining, from an image sensor, one or more images that represent an object, and determining a speed of the object based on the one or more images. The method also includes determining that the speed of the object exceeds a threshold speed and, based on determining that the speed of the object exceeds the threshold speed, determining a region of interest (ROI) of the image sensor expected to represent the object. The method further includes causing the image sensor to generate one or more ROI images using the ROI.

Smart sensor with region of interest capabilities

Example embodiments relate to camera arrangements for vehicular object detection and avoidance. An example system includes a vehicle and at least one camera of a first camera type attached to the vehicle. The system also includes a plurality of cameras of a second camera type attached to the vehicle. Further, the system includes a plurality of cameras of a third camera type attached to the vehicle. Moreover, the system includes a computing device communicatively coupled to the at least one camera of the first camera type, plurality of cameras of the second camera type, and plurality of cameras of the third camera type. The computing device is configured to identify objects located within a first range of distances from the vehicle, objects located within a second range of distances from the vehicle, and objects located within a third range of distances from the vehicle.

Camera arrangements for vehicular object detection and avoidance

A system includes an image sensor having a plurality of pixels that form a plurality of regions of interest (ROIs), and configured to operate at a frame rate higher than a threshold rate. The system also includes an image processing resource. The system further includes control circuitry configured to perform operations that include obtaining, from the image sensor, a full-resolution image of an environment. The full-resolution image contains each respective ROI of the plurality of ROIs. The operations also include selecting a particular ROI based on the full-resolution image, and detecting an object of interest in the particular ROI. The operations include determining a mode of operation by which subsequent image data generated by the particular ROI is to be processed. The operations further include processing, based on the mode of operation and the frame rate, the image data comprising a plurality of ROI images of the object of interest.

Smart sensor implementations of region of interest operating modes

A method includes obtaining, by a processing device, an impact analysis configuration related to an image sensor operation type for an autonomous vehicle (AV), receiving, by the processing device, image data from a sensing system including at least one image sensor of the AV, causing, by the processing device, fault detection to be performed based on the image data, causing, by the processing device, a fault notification to be generated using the impact analysis configuration, and sending, by the processing device to a data processing system of the AV, the fault notification to perform at least one action to address the fault notification. The fault notification includes a fault summary related to the image sensor operation type.

Autonomous vehicle sensor security, authentication and safety

Example embodiments relate to microlensing for real-time sensing of stray light. An example device includes an image sensor that includes a plurality of light-sensitive pixels. The device also includes a first lens positioned over a first subset of light-sensitive pixels selected from the plurality of light-sensitive pixels. Further, the device includes a controller. The controller is configured to determine a first angle of incidence of a first light signal detected by the first subset of light-sensitive pixels. The controller is also configured to, based on the first determined angle of incidence, determine an amount of stray light incident on the image sensor.

Microlensing for real-time sensing of stray light

An optical system may include a substrate and a plurality of silicon photomultipliers (SiPMs) monolithically integrated with the substrate. Each SiPM may include a plurality of single photon avalanche diodes (SPADs). The optical system also includes an aperture array having a plurality of apertures. The plurality of SiPMs and the aperture array are aligned so as to define a plurality of receiver channels. Each receiver channel includes a respective SiPM of the plurality of SiPMs optically coupled to a respective aperture of the plurality of apertures.

Monolithic silicon photomultiplier array

Described examples relate to an apparatus comprising a first sensor configured to scan an area of interest during a first time period and a second sensor configured to capture a plurality of images of a field of view. The apparatus may include at least one controller configured to receive the plurality of images captured by the second sensor, compare the timestamp information associated with at least one image of the plurality of images to at least one time period of the first time period, and select a base image from the plurality of images based on the comparison.

Systems, apparatus, and methods for generating enhanced images

A system includes a near-infrared (NIR) illuminator, an NIR image sensor, a light detection and ranging (LIDAR) device, and control circuitry configured to perform operations. The operations include causing the NIR illuminator to illuminate a portion of an environment, and obtaining, from the NIR image sensor, NIR image data representing the portion of the environment illuminated by the NIR illuminator. The operations also include detecting a retroreflector within the NIR image data and, based on detecting the retroreflector within the NIR image, determining a position of the retroreflector within the environment. The operations further include, based on the position of the retroreflector within the environment, adjusting at least one parameter of the LIDAR device in connection with scanning the retroreflector.

Detecting retroreflectors in NIR images to control LIDAR scan

The present disclosure relates to systems, vehicles, and methods for detecting optical defects in an optical path of a camera system. An example system may include an image sensor configured to provide images of a field of view via an optical path that extends through an optical window. The system also includes at least one phase-detection device and a controller. The controller is configured to execute instructions stored in the memory so as to carry out various operations, including receiving, from the image sensor, first pixel information indicative of an image of the field of view. The operations additionally include receiving, from the at least one phase-detection device, second pixel information indicative of a portion of the field of view. The operations yet further include determining, based on the first pixel information and the second pixel information, at least one optical defect associated with the optical path.

Aperture health monitoring mode

A system includes an image sensor having a plurality of pixels that form a plurality of regions of interest (ROIs), image processing resources, and a scheduler configured to perform operations including determining a priority level for a particular ROI of the plurality of ROIs based on a feature detected by one or more image processing resources of the image processing resources within initial image data associated with the particular ROI. The operations also include selecting, based on the feature detected within the initial image data, a particular image processing resource of the image processing resources by which subsequent image data generated by the particular ROI is to be processed. The operations further include inserting, based on the priority level, the subsequent image data into a processing queue of the particular image processing resource to schedule the subsequent image data for processing by the particular image processing resource.

Smart sensor scheduler

An apparatus includes an image sensor having a plurality of pixels that form regions of interest (ROIs), analog-to-digital converter (ADC) banks, and multiplexers. Each respective multiplexer is electrically connected to (i) a corresponding ADC bank and (ii) a corresponding subset of the ROIs. The apparatus also includes control circuitry configured to obtain a full-resolution image of an environment by electrically connecting, by way of the multiplexers, each respective ADC bank to the associated respective ROI. The control circuitry is also configured to select a particular ROI based on the full-resolution image and obtain a plurality of ROI images of the particular ROI by (i) electrically connecting, to the particular ROI, a first ADC bank associated with the particular ROI and a second ADC bank associated with another ROI and (ii) digitizing pixels of the particular ROI by way of parallel operation of the first and second ADC banks.

A system may include a first sensor of a first type and a second sensor of a second different type and having a detector. A field of view of the second sensor may be formed by a plurality of regions of interest (ROIs) defined by the detector. Control circuitry of the system may be configured to perform operations including obtaining, from the first sensor, first sensor data representing an environment, and determining, based on the first sensor data, information associated with a feature of interest within the environment. The operations may also include determining, based on the information, a particular ROI that corresponds to an expected position of the feature at a later time, obtaining a plurality of ROI sensor data from the particular ROI instead of obtaining full-resolution sensor data, and analyzing the plurality of ROI sensor data to determine one or more attributes of the feature.

Sensor region of interest selection based on multisensor data

A system includes an array of photodiode sensors positioned on an autonomous vehicle. Each photodiode sensor in the array of photodiode sensors is oriented in a different direction and configured to generate electrical current signals in response to detecting a flashing light in a surrounding environment of the autonomous vehicle. The system also includes a processor coupled to the array of photodiode sensors. The processor is configured to determine a location of a source of the flashing light relative to the autonomous vehicle based on electrical current signals from at least one photodiode sensor in the array of photodiode sensors. The processor is also configured to generate a command to maneuver the autonomous vehicle based on the location of the source relative to the autonomous vehicle.

Sensor for flashing light detection

The present disclosure relates to devices, light detection and ranging (lidar) systems, and vehicles involving solid-state, single photon detectors. An example device includes a substrate defining a primary plane and a plurality of photodetector cells disposed along the primary plane. The plurality of photodetector cells includes at least one large-area cell and at least one small-area cell. The large-area cell has a first area and the small-area cell has a second area and the first area is greater than the second area. The device also includes read out circuitry coupled to the plurality of photodetector cells. The read out circuitry is configured to provide an output signal based on incident light detected by the plurality of photodetector cells.

SiPM with cells of different sizes including at least one large-area cell is substantially centered along a substrate with respect to the optical axis of an aperture array

A method includes receiving, from a sensing system of an autonomous vehicle (AV), image data including first image data and second image data. The method further includes determining, for a frame, whether an amount of image data matching between the first image data and the second image data satisfies a first threshold condition, in response to determining that the amount of image data matching satisfies a first threshold condition, identifying the frame as invalid, determining whether a number of consecutive frames determined to be invalid satisfies a second threshold condition, and in response to determining that the number of consecutive frames determined to be invalid satisfies the second threshold condition, generating a notification that the sensing system is outputting invalid data.

Growing community of inventors

Milpitas, CA, United States of America

Nirav Dharia