Gm Global Technolgoy Operations, Inc.

Yao Hu

Xinyu Du

Hao Yu

Wende Zhang

Binbin Li

Yilu Zhang

Wen-Chiao Lin

Paul Edward Krajewski

Shiming Duan

Wei Tong

Shengbing Jiang

Arun Adiththan

Nicholas W Pinto

Kausalya Singuru

Xiaoyu Huang

Bradley R Frieberg

Yasen Hu

Mutasim A Salman

Rasoul Salehi

Zhongyu Yang

Guanlun He

Donald K Grimm

Bakhtiar Brian Litkouhi

Chih-Hung Yen

Chaitanya Sankavaram

Azeem Sarwar

Steven W Holland

Amir M Rahimi

Christopher Harold Knieper

Atul Nagose

Mohsen Khalili

Joshua J Sanchez

Rodolfo Valiente Romero

Thomas L Grime

Sidi Lu

Dana Warmsley

Xiaobo Guo

Faez Yahya

A computer-implemented method for aligning a sensor to reference coordinate system includes initiating a plurality of threads, each thread executes simultaneously and independent of each other. A first thread parses data received from the sensor and stores the parsed data in a data buffer. A second thread computes an alignment transformation using the parsed data to determine alignment between the sensor and the reference coordinate system. The computing includes checking that the data buffer contains at least predetermined amount of data. If at least the predetermined amount of data exists, an intermediate result is computed using the parsed data in the data buffer; otherwise, the second thread waits for the first thread to add more data to the data buffer. The second thread outputs the intermediate result into the data buffer. A third thread outputs the alignment transformation, in response to completion of alignment computations.

Vehicle-onboard computing architecture for sensor alignment

A vehicle, and a system and method of navigating the vehicle. The system includes a camera and a processor. The camera obtains an image of a road upon which the vehicle is moving. The processor is configured to extract a feature of the road from the image, perform a lane detection algorithm to detect a set of lane markers in the road using the image and the feature, and move the vehicle along the road by tracking the set of lane markers.

Mitigation strategies for lane marking misdetection

A system in a vehicle includes a lidar system to obtain lidar data in a lidar coordinate system, and processing circuitry to obtain the lidar data and localization data. The localization data indicates a location and orientation of the vehicle. The processing circuitry automatically determines an alignment state resulting in a lidar-to-vehicle transformation matrix that projects the lidar data from the lidar coordinate system to a vehicle coordinate system to provide lidar-to-vehicle data. The alignment state is determined using the localization data.

Automatic detection of lidar to vehicle alignment state using localization data

A vehicle includes a system performing a method of operating a camera for the vehicle. The camera is configured to capture an image. The processor is configured to query the camera to obtain an intrinsic parameter of the camera, estimate a unified parameter based on the intrinsic parameter of the camera, and perform at least one application on the image using the unified parameter.

Automatic unification of camera intrinsic model

A vision system for a vehicle includes a camera configured to generate one of distorted and undistorted images. An alignment system is configured to generate a region of interest (ROI) for the camera, generate a bird's eye view (BEV) homography to convert undistorted points in the ROI to BEV points, generate a BEV image based on the one of distorted and undistorted images and one of the BEV homography and an updated BEV homography, adaptively update the BEV homography to generate the updated BEV homography, detect current locations of features in the BEV image, predict future locations of the features for a next BEV image, compare the current locations of the features in the BEV image with prior predicted locations, and convert the BEV points in the BEV image to an undistorted points in a perspective undistorted image using an inverse of the updated BEV homography.

Vehicle vision system including camera to ground alignment using bird's eye view image

A vehicle includes a system that performs a method for aligning a first camera and a second camera to a reference frame of the vehicle. The first camera has a first coordinate system, and the second camera has a second coordinate system. The first camera obtains a first image. The processor is configured to obtain a first camera-to-ground transformation matrix for the first camera using the first image, wherein the first camera-to-ground transformation matrix relates the first coordinate system to a ground coordinate system, obtain a camera-to-camera transformation matrix between the first coordinate system of the first camera and the second coordinate system of the second camera, calculate a constructed camera-to-ground transformation matrix for the second camera using the first camera-to-ground transformation matrix and the camera-to-camera transformation matrix, and update the second coordinate system of the second camera using the constructed camera-to-ground transformation.

Camera to vehicle alignment with multiple camera adjustment

A system includes a platform operational to move through an environment, a navigation system operational to measure platform poses of the platform in the environment, a camera operational to generate images of the environment at multiple timestamps, and a computer. The computer is operational to receive a first image from the camera, receive a second image from the camera, estimate a view direction of the first camera pose based on a first and second platform poses and alignment parameters, generate a rendered portion of the environment with a neural radiance field technique based on the second image and the alignment parameters, generate a predicted image of the environment as observed along the view direction through the rendered portion of the environment, determine differences between the first image and the predicted image, and update one or more of the plurality of alignment parameters based on the one or more differences.

Neural radiance field based camera alignment

A system in a vehicle includes a lidar system to transmit incident light and receive reflections from one or more objects as a point cloud of points. The system also includes processing circuitry to identify feature points among the points of the point cloud, the feature points being horizontal feature points reflected from a horizontal surface or vertical feature points reflected from a vertical surface. The processing circuitry processes the point cloud by obtaining a normal vector corresponding to each of the points of the point cloud. The normal vector includes a first component associated with a first dimension, a second component associated with a second dimension, and a third component associated with a third dimension.

Identification of planar points in lidar point cloud obtained with vehicle lidar system

A system for verifying accuracy of an object determination made with a perception system for an object detected within a surrounding area of a vehicle. The system may include a semantic extraction module configured for generating semantic information for the object, a semantic generation module configured for generating a plurality of semantic captions based on the semantic information, a consistency evaluation module configured for generating consistency scores for the semantic captions, and a diagnostics module configured for verifying accuracy of the object determination based on the consistency scores.

Object detection verification for vehicle perception system

A camera misalignment detection system detecting a misalignment condition of one or more cameras for a vehicle includes one or more controllers that determine a set of matched pixel pairs, determine a feature matching ratio based on the set of matched pixel pairs, and calculate an alignment angle difference of the one or more cameras. In response to determining the feature matching ratio, the essential matrix inlier ratio, and the alignment angle difference each exceed respective threshold values, the controllers add the alignment angle difference to a queue including a sequence of historical alignment angle difference values. The controllers perform statistical filtering to determine a total number of historical alignment angle difference values within the queue that are inliers and determine a misalignment condition of the one or more cameras based on the total number of historical alignment angle difference values within the queue that are inliers.

Camera misalignment detection system for a vehicle

A system in a vehicle includes a lidar system to transmit incident light and receive reflections from one or more objects as a point cloud of points. The system also includes processing circuitry to identify planar points and to identify edge points of the point cloud. Each set of planar points forms a linear pattern and each edge point is between two sets of planar points, and the processing circuitry identifies each point of the points of the point cloud as being within a virtual beam among a set of virtual beams. Each virtual beam of the set of virtual beams representing a horizontal band of the point cloud.

Virtual beams for identification of edge and planar points in lidar point cloud obtained with vehicle lidar system

A vehicle, a system and a method of navigating the vehicle. The system includes a camera conveyed by the vehicle and a processor. The camera is configured to obtain an original image data file of an environment. The processor is configured to reduce the original image data file to obtain a reduced image data file and determine an alignment between a camera-centered coordinate system and a ground coordinate system using the reduced image data file.

System and method for online camera to ground alignment

A LIDAR-to-LIDAR alignment system includes a memory and an autonomous driving module. The memory stores first and second points based on outputs of first and second LIDAR sensors. The autonomous driving module performs a validation process to determine whether alignment of the LIDAR sensors satisfy an alignment condition. The validation process includes: aggregating the first and second points in a vehicle coordinate system to provide aggregated LIDAR points; based on the aggregated LIDAR points, performing (i) a first method including determining pitch and roll differences between the first and second LIDAR sensors, (ii) a second method including determining a yaw difference between the first and second LIDAR sensors, or (iii) point cloud registration to determine rotation and translation differences between the first and second LIDAR sensors; and based on results of the first method, the second method or the point cloud registration, determining whether the alignment condition is satisfied.

Online validation of LIDAR-to-LIDAR alignment and LIDAR-to-vehicle alignment

A LIDAR-to-vehicle alignment system includes a memory and alignment and autonomous driving modules. The memory stores points of data provided based on an output of one or more LIDAR sensors and localization data. The alignment module performs an alignment process including: based on the localization data; determining whether a host vehicle is turning; in response to the host vehicle turning; selecting a portion of the points of data; aggregating the selected portion to provide aggregated data; selecting targets based on the aggregated data; and based on the selected targets, iteratively reducing a loss value of a loss function to provide a resultant LIDAR-to-vehicle transformation matrix. The autonomous driving module: based on the resultant LIDAR-to-vehicle transformation matrix, converts at least the selected portion to at least one of vehicle coordinates or world coordinates to provide resultant data; and performs one or more autonomous driving operations based on the resultant data.

Aggregation-based LIDAR data alignment

A system for on-vehicle camera alignment monitoring includes an on-vehicle camera in communication with a controller. The controller monitors vehicle operating parameters and camera signal parameters, and captures an image file from the on-vehicle camera. A first level analysis of the image file, the vehicle operating parameters, and the camera signal parameters is executed to detect dynamic conditions and image feature parameters that affect camera alignment. An error with one of the dynamic conditions or the image feature parameters that affects the camera alignment is detected. A second level analysis of the camera signal parameters is executed to identify a root cause indicating one of the dynamic conditions or the image feature parameters that affects the camera alignment based upon the error. A camera alignment-related fault is detected based upon the root cause, and vehicle operation is controlled based upon the camera alignment-related fault.

Growing community of inventors

Sterling Heights, MI, United States of America

Yao Hu