Waymo LLC

Nio Technology (anhui) Co., Ltd.

Nio Usa, Inc.

Xu Chen

Ruichi Yu

Tong Lin

Hiu Hong Yu

Zhenxiang Jian

Veera Ganesh Yalla

Hang Zhao

Kang Li

Ashwath Aithal

Shiwei Sheng

Ruoyu Tan

Huazeng Deng

Ajaya H S Rao

Farzad Cyrus Foroughi Abari

Congcong Li

Alper Ayvaci

Jiyang Gao

Henrik Kretzschmar

Nichola Abdo

Ruizhongtai Qi

Zhongkui Wang

Waylon Y Chen

Dillon Cower

Andre Michelin

Luming Tang

Chieh-En Tsai

Chang Gao

Longlong Jing

Antonio Antonellis Rufo

The described aspects and implementations enable fast and accurate object identification in autonomous vehicle (AV) applications by combining radar data with camera images. In one implementation, disclosed is a method and a system to perform the method that includes obtaining a radar image of a first hypothetical object in an environment of the AV, obtaining a camera image of a second hypothetical object in the environment of the AV, and processing the radar image and the camera image using one or more machine-learning models MLMs to obtain a prediction measure representing a likelihood that the first hypothetical object and the second hypothetical object correspond to a same object in the environment of the AV.

Association of camera images and radar data in autonomous vehicle applications

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for predicting three-dimensional object locations from images. One of the methods includes obtaining a sequence of images that comprises, at each of a plurality of time steps, a respective image that was captured by a camera at the time step; generating, for each image in the sequence, respective pseudo-lidar features of a respective pseudo-lidar representation of a region in the image that has been determined to depict a first object; generating, for a particular image at a particular time step in the sequence, image patch features of the region in the particular image that has been determined to depict the first object; and generating, from the respective pseudo-lidar features and the image patch features, a prediction that characterizes a location of the first object in a three-dimensional coordinate system at the particular time step in the sequence.

Three-dimensional location prediction from images

The described aspects and implementations enable fast and accurate verification of radar detection of objects in autonomous vehicle (AV) applications using combined processing of radar data and camera images. In one implementation, disclosed is a method and a system to perform the method that includes obtaining a radar data characterizing intensity of radar reflections from an environment of the AV, identifying, based on the radar data, a candidate object, obtaining a camera image depicting a region where the candidate object is located, and processing the radar data and the camera image using one or more machine-learning models to obtain a classification measure representing a likelihood that the candidate object is a real object.

Identification of spurious radar detections in autonomous vehicle applications

The described aspects and implementations enable efficient calibration of a sensing system of a vehicle. In one implementation, disclosed is a method and a system to perform the method, the system including the sensing system configured to obtain a plurality of images associated with a corresponding time of a plurality of times. The system further includes a data processing system operatively coupled to the sensing system and configured to generate a plurality of sets of feature tensors (FTs) associated with one or more objects of the environment depicted in a respective image. The data processing system is further to obtain a combined FT and process the combined FT using a neural network to identify one or more tracks characterizing motion of a respective object.

End-to-end object tracking using neural networks with attention

Example embodiments relate to techniques for three dimensional (3D) object detection and localization. A computing system may cause a radar unit to transmit radar signals and receive radar reflections relative to an environment of a vehicle. Based on the radar reflections, the computing system may determine a heading and a range for a nearby object. The computing system may also receive an image depicting a portion of the environment that includes the object from a vehicle camera and remove peripheral areas of the image to generate an image patch that focuses upon the object based on the heading and the range for the object. The image patch and the heading and the range for the object can be provided as inputs into a neural network that provides output parameters corresponding to the object, which can be used to control the vehicle.

Methods and systems for three dimensional object detection and localization

A calibration system for multi-sensor extrinsic calibration in a vehicle includes one or more calibration targets provided around an external environment within a threshold distance of the vehicle. Each of the one or more calibration targets includes a combination of sensor targets configured to be measured by and used for calibrating a pair of sensors selected from the group consisting of a first sensor, a second sensor or a third sensor. The system also includes a vehicle placement section configured to accommodate the vehicle on the vehicle placement section for detection of the one or more calibration targets.

High precision multi-sensor extrinsic calibration via production line and mobile station

Embodiments of the present disclosure are directed to providing scene perception display requiring reduced processing capabilities. Sensor data indicative of one or more targets from an imaging and ranging subsystem, location data from a positioning subsystem defining a geographical location of the imaging and ranging subsystem and orientation data from an orientation subsystem defining an orientation of the imaging and ranging subsystem are received. Doppler point cloud data is generated based on Doppler information and point cloud data from the sensor data, the location data and the orientation data. The targets are classified as either a static or dynamic target. Afterwards, the Doppler point cloud data is filtered by removing either the dynamic or the static targets from the Doppler point cloud data. The Doppler point cloud data of either the dynamic or the static targets are further processed and the further processed Doppler point cloud data is rendered.

Scene perception using coherent doppler LiDAR

Embodiments of the present disclosure are directed to a method for object detection. The method includes receiving sensor data indicative of one or more objects for each of a camera subsystem, a LiDAR subsystem, and an imaging RADAR subsystem. The sensor data is received simultaneously and within one frame for each of the subsystems. The method also includes extracting one or more feature representations of the objects from camera image data, LiDAR point cloud data and imaging RADAR point cloud data and generating image feature maps, LiDAR feature maps and imaging RADAR feature maps. The method further includes combining the image feature maps, the LiDAR feature maps and the imaging RADAR feature maps to generate merged feature maps and generating object classification, object position, object dimensions, object heading and object velocity from the merged feature maps.

Single frame 4D detection using deep fusion of camera image, imaging RADAR and LiDAR point cloud

Embodiments of the present disclosure are directed to calibrating an imaging and ranging subsystem. Sensor data indicative of one or more targets from the imaging and ranging subsystem, location data defining a geographical location of the imaging and ranging subsystem, orientation data defining an orientation of the imaging and ranging subsystem and stored translation and rotation values of the imaging and ranging subsystem are received. Estimated Doppler values for the target are provided by the sensor data and theoretical Doppler values for the targets are also calculated. The estimated Doppler values are compared to the theoretical Doppler values to determine if calibration of the imaging and ranging subsystem is required. If calibration is necessary, correction translation and correction rotation values are calculated in order to calculate updated translation and rotation values used to calibrate the imaging and ranging subsystem.

Real-time online calibration of coherent doppler lidar systems on vehicles

Extrinsic calibration of a Light Detection and Ranging (LiDAR) sensor and a camera can comprise constructing a first plurality of reconstructed calibration targets in a three-dimensional space based on physical calibration targets detected from input from the LiDAR and a second plurality of reconstructed calibration targets in the three-dimensional space based on physical calibration targets detected from input from the camera. Reconstructed calibration targets in the first and second plurality of reconstructed calibration targets can be matched and a six-degree of freedom rigid body transformation of the LiDAR and camera can be computed based on the matched reconstructed calibration targets. A projection of the LiDAR to the camera can be computed based on the computed six-degree of freedom rigid body transformation.

Automatic autonomous vehicle and robot LiDAR-camera extrinsic calibration

Methods and systems include localization of a vehicle localize precisely and in near real-time. As described, localization of a vehicle using a Global Navigation Satellite System (GNSS) can comprise receiving a signal from each of a plurality of satellites of a GNSS constellation and receiving input from one or more sensors of the vehicle. The input from the sensors can indicate current physical surroundings of the vehicle. A model of the current physical surrounding of the vehicle can be generated based on the input from the one or more sensors of the vehicle. One or more multipath signals received from the plurality of satellites can be mitigated based on the model and the vehicle can be localized using the received signals from the plurality of satellites of the GNSS constellation and based on the mitigation of the one or more multipath signals.

Growing community of inventors

Livermore, CA, United States of America

Xu Chen