Waymo LLC

Apple Inc.

Lucian Ion

Nirav Dharia

Vlad Constantin Cardei

Giulia Guidi

Choon Ping Chng

Cheng-Han Wu

Carl Warren Craddock

Ralph Hamilton Shepard

Benjamin Frevert

Xiaoying He

Evan C Cull

Ryan M Geiss

Daniel E Potter

Xiaofeng Tang

Jack E Graves

Ajay Chandra Venkata Gummalla

Zhinan Xu

Pieter Kapsenberg

Shashank Sharma

Clarisse Mazuir

Damien Kelly

Peyman Milanfar

Joseph Patrick Warga

Bryce Jason Remesch

Budhadipta Dan

Christy Fernandez Cull

James J Reuther

Filip Ilievski

Mira S Misra

Matisse J Milovich

Mahesh Krishnamurthi

Irene Perali

Ignacio Garcia-Dorado

Christopher M Miller

Sheila P Nabanja

Bartlomiej Wronski

Vlad Cardei

Example embodiments relate to camera arrangements for vehicular object detection and avoidance. An example system includes a vehicle and at least one camera of a first camera type attached to the vehicle. The system also includes a plurality of cameras of a second camera type attached to the vehicle. Further, the system includes a plurality of cameras of a third camera type attached to the vehicle. Moreover, the system includes a computing device communicatively coupled to the at least one camera of the first camera type, plurality of cameras of the second camera type, and plurality of cameras of the third camera type. The computing device is configured to identify objects located within a first range of distances from the vehicle, objects located within a second range of distances from the vehicle, and objects located within a third range of distances from the vehicle.

Camera arrangements for vehicular object detection and avoidance

Example embodiments relate to foveated imagers for automotive applications. An example embodiment includes a device. The device includes a rotationally symmetric foveated lens. The rotationally symmetric foveated lens is configured to receive light from an environment. The rotationally symmetric foveated lens is also configured to produce an image at an image plane based on the received light. The device also includes an image sensor having an associated image sensor resolution. The image sensor is positioned at the image plane and configured to capture an image having an associated field of view of the environment. Based on a distortion profile of the rotationally symmetric foveated lens and the image sensor resolution, the captured image exhibits a first angular optical resolution in a central region of the field of view and a second angular optical resolution in a peripheral region of the field of view.

Foveated imager for automotive applications

A system includes an image sensor having a plurality of pixels that form a plurality of regions of interest (ROIs), and configured to operate at a frame rate higher than a threshold rate. The system also includes an image processing resource. The system further includes control circuitry configured to perform operations that include obtaining, from the image sensor, a full-resolution image of an environment. The full-resolution image contains each respective ROI of the plurality of ROIs. The operations also include selecting a particular ROI based on the full-resolution image, and detecting an object of interest in the particular ROI. The operations include determining a mode of operation by which subsequent image data generated by the particular ROI is to be processed. The operations further include processing, based on the mode of operation and the frame rate, the image data comprising a plurality of ROI images of the object of interest.

Smart sensor implementations of region of interest operating modes

A system is provided that includes an image sensor coupled to a vehicle, and control circuitry configured to perform operations including receiving, from the image sensor, an input stream comprising high dynamic range (HDR) image data associated with an environment of the vehicle, and processing the input stream at the vehicle by applying a global tone mapping, followed by offline image processing that can include applying a local tone mapping to the globally tone mapped images of the same input stream.

Systems and methods for a tone mapper solution for an autonomous driving system

Described examples relate to an apparatus comprising a memory for storing image frames and at least one processor. The at least one processor may be configured to receive a plurality of image frames from an image capture device and downsize each of the plurality image frames to generate a plurality of versions of each image frame at a plurality of different sizes. The at least one processor may also be configured to determine alignment information for a first version of a first image frame. The alignment information may include a first alignment vector for identifying image data in a first version of a second image frame that corresponds to image data in the first version of the first image frame. Further, the at least one processor may be configured to determine a first initial alignment vector for identifying image data in a first version of a third image frame based on at least the first alignment vector.

Systems, methods, and apparatus for aligning image frames

Example embodiments relate to microlensing for real-time sensing of stray light. An example device includes an image sensor that includes a plurality of light-sensitive pixels. The device also includes a first lens positioned over a first subset of light-sensitive pixels selected from the plurality of light-sensitive pixels. Further, the device includes a controller. The controller is configured to determine a first angle of incidence of a first light signal detected by the first subset of light-sensitive pixels. The controller is also configured to, based on the first determined angle of incidence, determine an amount of stray light incident on the image sensor.

Microlensing for real-time sensing of stray light

Example embodiments described herein involve a system for testing a light-emitting module. The light-emitting module may include a mounting platform configured to hold a light-emitting module for a camera. The mounting platform may also be configured to rotate. The system may further include a housing holding a plurality of photodiodes arranged in an array over at least a 90 degree arc of a hemisphere. The system may also include a controller configured to control the photodiodes and the rotation of the mounting platform.

Infrared light module uniformity rotational test module

A method includes obtaining multiple images captured by pixel sensors of an image sensor, analyzing, using neural network circuitry integrated in the image sensor, the multiple images for object detection, generating, for each of the multiple images using the neural network circuitry integrated in the image sensor, neural network output data related to results of the analysis of the multiple images for object detection, and transmitting, from the image sensor, the neural network output data for each of the multiple images and image data for a subset of the multiple images instead of image data of each of the multiple images.

High bandwidth camera data transmission

Disclosed are devices and methods that may be used for the calibration of sensors of a vehicle. A calibration system disclosed herein includes a trigger device configured to detect a first sensing event. The calibration system further includes a plurality of sensor targets. Additionally, the calibration system includes a processor configured to reveal the plurality of sensor targets within a given region in response to the first sensing event. The calibration system may be configured to detect light from a LIDAR striking a light sensors. In response to detecting the light from the LIDAR, a plurality of lights, both visible and infrared, may be illuminated to correspond with a position and timing of the LIDAR pulse.

Multi-sensor synchronization measurement device

A system includes an image sensor having a plurality of pixels that form a plurality of regions of interest (ROIs), image processing resources, and a scheduler configured to perform operations including determining a priority level for a particular ROI of the plurality of ROIs based on a feature detected by one or more image processing resources of the image processing resources within initial image data associated with the particular ROI. The operations also include selecting, based on the feature detected within the initial image data, a particular image processing resource of the image processing resources by which subsequent image data generated by the particular ROI is to be processed. The operations further include inserting, based on the priority level, the subsequent image data into a processing queue of the particular image processing resource to schedule the subsequent image data for processing by the particular image processing resource.

Smart sensor scheduler

Systems, apparatus, and methods are presented for event detection. One method may include receiving event data from an event capture device while the event capture device is in a stationary state. The event capture device may be coupled to a vehicle. The method may also include generating a command and sending the command to the event capture device. The command is configured to cause the event capture device to change from the stationary state to a non-stationary state.

Systems, methods, and apparatus for event detection

A image sensor includes a first integrated circuit layer including pixel sensors that are grouped based on position into pixel sensor groups, a second integrated circuit layer in electrical communication with the first integrated circuit layer, the second integrated circuit layer including image processing circuitry groups that are configured to each receive pixel information from a corresponding pixel sensor group, the image processing circuitry groups further configured to perform image processing operations on the pixel information to provide processed pixel information during operation of the image sensor, a third integrated circuit layer in electrical communication with the second integrated circuit layer, and the third integrated circuit layer including neural network circuitry groups that are configured to each receive the processed pixel information from a corresponding image processing circuitry group and perform analysis for object detection on the processed pixel information during operation of the image sensor.

Image sensor architecture

An apparatus includes an image sensor having a plurality of pixels that form regions of interest (ROIs), analog-to-digital converter (ADC) banks, and multiplexers. Each respective multiplexer is electrically connected to (i) a corresponding ADC bank and (ii) a corresponding subset of the ROIs. The apparatus also includes control circuitry configured to obtain a full-resolution image of an environment by electrically connecting, by way of the multiplexers, each respective ADC bank to the associated respective ROI. The control circuitry is also configured to select a particular ROI based on the full-resolution image and obtain a plurality of ROI images of the particular ROI by (i) electrically connecting, to the particular ROI, a first ADC bank associated with the particular ROI and a second ADC bank associated with another ROI and (ii) digitizing pixels of the particular ROI by way of parallel operation of the first and second ADC banks.

Smart sensor with region of interest capabilities

A system may include a first sensor of a first type and a second sensor of a second different type and having a detector. A field of view of the second sensor may be formed by a plurality of regions of interest (ROIs) defined by the detector. Control circuitry of the system may be configured to perform operations including obtaining, from the first sensor, first sensor data representing an environment, and determining, based on the first sensor data, information associated with a feature of interest within the environment. The operations may also include determining, based on the information, a particular ROI that corresponds to an expected position of the feature at a later time, obtaining a plurality of ROI sensor data from the particular ROI instead of obtaining full-resolution sensor data, and analyzing the plurality of ROI sensor data to determine one or more attributes of the feature.

Growing community of inventors

Santa Clara, CA, United States of America

Lucian Ion