Gm Global Technolgoy Operations, Inc.

Hao Yu

Wende Zhang

Xinyu Du

Yao Hu

Binbin Li

Yousef A Omar

Mohsen Khalili

Guanlun He

Robert D Sims, Iii

Nathaniel W Hart

Farui Peng

Mark J Wolski

Hongtao Wang

Xiaobo Guo

A sensor alignment method including detecting, detecting an object within a first sensor field of view and a second sensor field of view, detecting a plurality of depth point clouds of the first field of view in response to the object being stationary, the host vehicle being stationary, and the distance between the object and the host vehicle being less than a threshold distance, aggregating the plurality of depth point clouds into an aggregated depth point cloud, detecting a first location of an edge of the object in response to the aggregated depth point cloud, detecting a second location of the edge of the object using an edge detection algorithm on the image, generating a lidar to camera alignment in response to a difference between the first location and the second location.

Robust lidar-to-camera sensor alignment

A computer-implemented method for aligning a sensor to reference coordinate system includes initiating a plurality of threads, each thread executes simultaneously and independent of each other. A first thread parses data received from the sensor and stores the parsed data in a data buffer. A second thread computes an alignment transformation using the parsed data to determine alignment between the sensor and the reference coordinate system. The computing includes checking that the data buffer contains at least predetermined amount of data. If at least the predetermined amount of data exists, an intermediate result is computed using the parsed data in the data buffer; otherwise, the second thread waits for the first thread to add more data to the data buffer. The second thread outputs the intermediate result into the data buffer. A third thread outputs the alignment transformation, in response to completion of alignment computations.

Vehicle-onboard computing architecture for sensor alignment

Methods, apparatus, and systems for sensor alignment include acquiring a translational vector, a first calibration point location and a second calibration point location, determining an expected rotational orientation in response to the translational vector, the first calibration point location and the second calibration point location, capturing an image of the a first calibration point and the second calibration point, determining a first position of the first calibration point and a second position of the second calibration point in response to the image, calculating a calculated rotational orientation in response to the first position of the first calibration point and the second position of the second calibration point, determining a calibration value in response to the calculated rotational orientation, storing the calibration value and controlling a vehicle in response to the calibration value and a subsequent image.

Sensor angular alignment

A system in a vehicle includes a lidar system to obtain lidar data in a lidar coordinate system, and processing circuitry to obtain the lidar data and localization data. The localization data indicates a location and orientation of the vehicle. The processing circuitry automatically determines an alignment state resulting in a lidar-to-vehicle transformation matrix that projects the lidar data from the lidar coordinate system to a vehicle coordinate system to provide lidar-to-vehicle data. The alignment state is determined using the localization data.

Automatic detection of lidar to vehicle alignment state using localization data

A vehicle includes a system performing a method of operating a camera for the vehicle. The camera is configured to capture an image. The processor is configured to query the camera to obtain an intrinsic parameter of the camera, estimate a unified parameter based on the intrinsic parameter of the camera, and perform at least one application on the image using the unified parameter.

Automatic unification of camera intrinsic model

A vision system for a vehicle includes a camera configured to generate one of distorted and undistorted images. An alignment system is configured to generate a region of interest (ROI) for the camera, generate a bird's eye view (BEV) homography to convert undistorted points in the ROI to BEV points, generate a BEV image based on the one of distorted and undistorted images and one of the BEV homography and an updated BEV homography, adaptively update the BEV homography to generate the updated BEV homography, detect current locations of features in the BEV image, predict future locations of the features for a next BEV image, compare the current locations of the features in the BEV image with prior predicted locations, and convert the BEV points in the BEV image to an undistorted points in a perspective undistorted image using an inverse of the updated BEV homography.

Vehicle vision system including camera to ground alignment using bird's eye view image

A vehicle includes a system that performs a method for aligning a first camera and a second camera to a reference frame of the vehicle. The first camera has a first coordinate system, and the second camera has a second coordinate system. The first camera obtains a first image. The processor is configured to obtain a first camera-to-ground transformation matrix for the first camera using the first image, wherein the first camera-to-ground transformation matrix relates the first coordinate system to a ground coordinate system, obtain a camera-to-camera transformation matrix between the first coordinate system of the first camera and the second coordinate system of the second camera, calculate a constructed camera-to-ground transformation matrix for the second camera using the first camera-to-ground transformation matrix and the camera-to-camera transformation matrix, and update the second coordinate system of the second camera using the constructed camera-to-ground transformation.

Camera to vehicle alignment with multiple camera adjustment

A system includes a platform operational to move through an environment, a navigation system operational to measure platform poses of the platform in the environment, a camera operational to generate images of the environment at multiple timestamps, and a computer. The computer is operational to receive a first image from the camera, receive a second image from the camera, estimate a view direction of the first camera pose based on a first and second platform poses and alignment parameters, generate a rendered portion of the environment with a neural radiance field technique based on the second image and the alignment parameters, generate a predicted image of the environment as observed along the view direction through the rendered portion of the environment, determine differences between the first image and the predicted image, and update one or more of the plurality of alignment parameters based on the one or more differences.

Neural radiance field based camera alignment

A system in a vehicle includes a lidar system to transmit incident light and receive reflections from one or more objects as a point cloud of points. The system also includes processing circuitry to identify feature points among the points of the point cloud, the feature points being horizontal feature points reflected from a horizontal surface or vertical feature points reflected from a vertical surface. The processing circuitry processes the point cloud by obtaining a normal vector corresponding to each of the points of the point cloud. The normal vector includes a first component associated with a first dimension, a second component associated with a second dimension, and a third component associated with a third dimension.

Identification of planar points in lidar point cloud obtained with vehicle lidar system

A camera misalignment detection system detecting a misalignment condition of one or more cameras for a vehicle includes one or more controllers that determine a set of matched pixel pairs, determine a feature matching ratio based on the set of matched pixel pairs, and calculate an alignment angle difference of the one or more cameras. In response to determining the feature matching ratio, the essential matrix inlier ratio, and the alignment angle difference each exceed respective threshold values, the controllers add the alignment angle difference to a queue including a sequence of historical alignment angle difference values. The controllers perform statistical filtering to determine a total number of historical alignment angle difference values within the queue that are inliers and determine a misalignment condition of the one or more cameras based on the total number of historical alignment angle difference values within the queue that are inliers.

Camera misalignment detection system for a vehicle

A system in a vehicle includes a lidar system to transmit incident light and receive reflections from one or more objects as a point cloud of points. The system also includes processing circuitry to identify planar points and to identify edge points of the point cloud. Each set of planar points forms a linear pattern and each edge point is between two sets of planar points, and the processing circuitry identifies each point of the points of the point cloud as being within a virtual beam among a set of virtual beams. Each virtual beam of the set of virtual beams representing a horizontal band of the point cloud.

Virtual beams for identification of edge and planar points in lidar point cloud obtained with vehicle lidar system

A vehicle, a system and a method of navigating the vehicle. The system includes a camera conveyed by the vehicle and a processor. The camera is configured to obtain an original image data file of an environment. The processor is configured to reduce the original image data file to obtain a reduced image data file and determine an alignment between a camera-centered coordinate system and a ground coordinate system using the reduced image data file.

System and method for online camera to ground alignment

A LIDAR-to-LIDAR alignment system includes a memory and an autonomous driving module. The memory stores first and second points based on outputs of first and second LIDAR sensors. The autonomous driving module performs a validation process to determine whether alignment of the LIDAR sensors satisfy an alignment condition. The validation process includes: aggregating the first and second points in a vehicle coordinate system to provide aggregated LIDAR points; based on the aggregated LIDAR points, performing (i) a first method including determining pitch and roll differences between the first and second LIDAR sensors, (ii) a second method including determining a yaw difference between the first and second LIDAR sensors, or (iii) point cloud registration to determine rotation and translation differences between the first and second LIDAR sensors; and based on results of the first method, the second method or the point cloud registration, determining whether the alignment condition is satisfied.

Online validation of LIDAR-to-LIDAR alignment and LIDAR-to-vehicle alignment

A LIDAR-to-vehicle alignment system includes a memory and alignment and autonomous driving modules. The memory stores points of data provided based on an output of one or more LIDAR sensors and localization data. The alignment module performs an alignment process including: based on the localization data; determining whether a host vehicle is turning; in response to the host vehicle turning; selecting a portion of the points of data; aggregating the selected portion to provide aggregated data; selecting targets based on the aggregated data; and based on the selected targets, iteratively reducing a loss value of a loss function to provide a resultant LIDAR-to-vehicle transformation matrix. The autonomous driving module: based on the resultant LIDAR-to-vehicle transformation matrix, converts at least the selected portion to at least one of vehicle coordinates or world coordinates to provide resultant data; and performs one or more autonomous driving operations based on the resultant data.

Aggregation-based LIDAR data alignment

A system for determining calibrated camera extrinsic parameters for an autonomous vehicle includes a camera mounted to the autonomous vehicle collecting image data including a plurality of image frames. The system also includes one or more automated driving controllers in electronic communication with the camera that executes instructions to determine a vehicle pose estimate based on position and movement of the autonomous vehicle by a localization algorithm. The one or more automated driving controllers determine the calibrated camera extrinsic parameters based on three dimensional coordinates for specific feature points of interests corresponding to two sequential image frames, the specific feature points of interests corresponding to the two sequential image frames, and the camera pose corresponding to the two sequential image frames.

Growing community of inventors

Troy, MI, United States of America

Hao Yu