Gm Global Technology Operations LLC

Carnegie Mellon University

Wende Zhang

Jinsong Wang

Bakhtiar Brian Litkouhi

Hao Yu

Kent S Lybecker

Sanjeev Manubhai Naik

Xinyu Du

Yao Hu

Ryan M Frakes

Shuqing Zeng

Fan Bai

Jeffrey S Piasecki

Qi Wu

Donald K Grimm

Esther Anderson

Qingrong Zhao

Binbin Li

Tsuhan Chen

Travis S Hester

Varsha K Sadekar

Mark J Wolski

Xiujie Gao

Mohannad F Murad

Wei Tong

Dan Levi

Kamran Ali

Fernando De La Torre

Mark Edward Shields

Debbie E Nachtegall

Mark T Gaylord

Yousef A Omar

Timothy J Talty

Robert Alan Dziurda

Vijayakumar Bhagavatula

Qi Zhang

Caroline Chung

Xi Ju

Jiang L Du

Zhencheng Hu

Adrienne M Williamson

Timothy J Canner

Nancy L Johnson

Thomas A Seder

Paul Edward Krajewski

Upali Priyantha Mudalige

Xiaosong Huang

Charles A Green

Norman Joseph Weigert

Jin-Woo Lee

Duane S Carper

Allan K Lewis

Sai Vishnu Aluru

Xin Yu

Joseph G Machak

Robert D Sims, Iii

Chaminda Basnayake

Michael A Wuergler

John Sergakis

Chuan Li

Cem U Saraydar

Jeffrey A Bozeman

Michael T Chaney, Jr

Jace C Stokes

Brunno L Moretti

Mohsen Khalili

Christopher A Stanek

Kevin K Hoang

Dong Huang

Xuehan Xiong

Veronique C Prinet

Franscisco Vicente

Kevin Peterson

Christopher Paul Urmson

Paul Andrew Justin

Alma Bushi

Gary V Bandurski

Xinhua Xiao

Jie Yin

Francisco Vicente

Zehua Huang

James Clem

Vernon Hole

A system includes a platform operational to move through an environment, a navigation system operational to measure platform poses of the platform in the environment, a camera operational to generate images of the environment at multiple timestamps, and a computer. The computer is operational to receive a first image from the camera, receive a second image from the camera, estimate a view direction of the first camera pose based on a first and second platform poses and alignment parameters, generate a rendered portion of the environment with a neural radiance field technique based on the second image and the alignment parameters, generate a predicted image of the environment as observed along the view direction through the rendered portion of the environment, determine differences between the first image and the predicted image, and update one or more of the plurality of alignment parameters based on the one or more differences.

Neural radiance field based camera alignment

A system in a vehicle includes a lidar system to transmit incident light and receive reflections from one or more objects as a point cloud of points. The system also includes processing circuitry to identify feature points among the points of the point cloud, the feature points being horizontal feature points reflected from a horizontal surface or vertical feature points reflected from a vertical surface. The processing circuitry processes the point cloud by obtaining a normal vector corresponding to each of the points of the point cloud. The normal vector includes a first component associated with a first dimension, a second component associated with a second dimension, and a third component associated with a third dimension.

Identification of planar points in lidar point cloud obtained with vehicle lidar system

A system in a vehicle includes a lidar system to transmit incident light and receive reflections from one or more objects as a point cloud of points. The system also includes processing circuitry to identify planar points and to identify edge points of the point cloud. Each set of planar points forms a linear pattern and each edge point is between two sets of planar points, and the processing circuitry identifies each point of the points of the point cloud as being within a virtual beam among a set of virtual beams. Each virtual beam of the set of virtual beams representing a horizontal band of the point cloud.

Virtual beams for identification of edge and planar points in lidar point cloud obtained with vehicle lidar system

A LIDAR-to-LIDAR alignment system includes a memory and an autonomous driving module. The memory stores first and second points based on outputs of first and second LIDAR sensors. The autonomous driving module performs a validation process to determine whether alignment of the LIDAR sensors satisfy an alignment condition. The validation process includes: aggregating the first and second points in a vehicle coordinate system to provide aggregated LIDAR points; based on the aggregated LIDAR points, performing (i) a first method including determining pitch and roll differences between the first and second LIDAR sensors, (ii) a second method including determining a yaw difference between the first and second LIDAR sensors, or (iii) point cloud registration to determine rotation and translation differences between the first and second LIDAR sensors; and based on results of the first method, the second method or the point cloud registration, determining whether the alignment condition is satisfied.

Online validation of LIDAR-to-LIDAR alignment and LIDAR-to-vehicle alignment

A LIDAR-to-vehicle alignment system includes a memory and alignment and autonomous driving modules. The memory stores points of data provided based on an output of one or more LIDAR sensors and localization data. The alignment module performs an alignment process including: based on the localization data; determining whether a host vehicle is turning; in response to the host vehicle turning; selecting a portion of the points of data; aggregating the selected portion to provide aggregated data; selecting targets based on the aggregated data; and based on the selected targets, iteratively reducing a loss value of a loss function to provide a resultant LIDAR-to-vehicle transformation matrix. The autonomous driving module: based on the resultant LIDAR-to-vehicle transformation matrix, converts at least the selected portion to at least one of vehicle coordinates or world coordinates to provide resultant data; and performs one or more autonomous driving operations based on the resultant data.

Aggregation-based LIDAR data alignment

A system for on-vehicle camera alignment monitoring includes an on-vehicle camera in communication with a controller. The controller monitors vehicle operating parameters and camera signal parameters, and captures an image file from the on-vehicle camera. A first level analysis of the image file, the vehicle operating parameters, and the camera signal parameters is executed to detect dynamic conditions and image feature parameters that affect camera alignment. An error with one of the dynamic conditions or the image feature parameters that affects the camera alignment is detected. A second level analysis of the camera signal parameters is executed to identify a root cause indicating one of the dynamic conditions or the image feature parameters that affects the camera alignment based upon the error. A camera alignment-related fault is detected based upon the root cause, and vehicle operation is controlled based upon the camera alignment-related fault.

On-vehicle camera alignment monitoring system

Systems and methods for a vehicle are provided. In one embodiment, a method includes: receiving image data defining a plurality of images associated with an environment of the vehicle; determining, by a processor, feature points within at least one image of the plurality of images; selecting, by the processor, a subset of the feature points as ground points based on a fixed two dimensional image road mask and a three dimensional region; determining, by the processor, a ground plane based on the subset of feature points; determining, by the processor, a ground normal vector from the ground plane; determining, by the processor, a camera to ground alignment value based on the ground normal vector; and generating, by the processor, second image data based on the camera to ground alignment value.

Methods and systems for camera to ground alignment

Methods and systems for a vehicle are provided. In one embodiment, the method includes: receiving image data defining a plurality of images associated with an environment of the vehicle; determining, by a processor, feature points within at least one image of the plurality of images; selecting, by the processor, a subset of the feature points as ground points; determining, by the processor, a ground plane based on the subset of feature points; determining, by the processor, a ground normal vector from the ground plane; determining, by the processor, the ground normal vector based on a sliding widow method; determining, by the processor, a camera to ground alignment value based on the ground normal vector; and generating, by the processor, second image data based on the camera to ground alignment value.

A vehicle control system including a spatial monitoring system includes on-vehicle cameras that capture images, from which are recovered a plurality of three-dimensional points. A left ground plane normal vector is determined for a left image, a center ground plane normal vector is determined for a front image, and a right ground plane normal vector is determined for a right image. A first angle difference between the left ground plane normal vector and the center ground plane normal vector is determined, and a second angle difference between the right ground plane normal vector and the center ground plane normal vector is determined. An uneven ground surface is determined based upon one of the first angle difference or the second angle difference, and an alignment compensation factor for the left camera or the right camera is determined. A bird's eye view image is determined based upon the alignment compensation factor.

On-vehicle spatial monitoring system

A scan matching and radar pose estimator for determining a final radar pose for an autonomous vehicle includes an automated driving controller that is instructed to determine a hyper-local submap based on a predefined number of consecutive aggregated filtered data point cloud scans and associated pose estimates. The automated driving controller determines an initial estimated pose by aligning a latest aggregated filtered data point cloud scan with the most recent hyper-local submap based on an iterative closest point (ICP) alignment algorithm. The automated driving controller determines a pose graph based on the most recent hyper-local submap and neighboring radar point cloud scans, and executes a multi-view non-linear ICP algorithm to adjust initial estimated poses corresponding to the neighboring radar point cloud scans in a moving window fashion to determine a locally adjusted pose.

Scan matching and radar pose estimator for an autonomous vehicle based on hyper-local submaps

Systems and methods for a vehicle are provided. In one embodiment, a method includes: receiving image data defining a plurality of images associated with an environment of the vehicle; receiving vehicle data indicating a velocity of the vehicle, wherein the vehicle data is associated with the image data; determining, by a processor, feature points within at least one image based on the vehicle data and a three dimensional projection method; selecting, by the processor, a subset of the feature points as ground points; determining, by the processor, a ground plane based on the subset of feature points; determining, by the processor, a ground normal vector from the ground plane; determining, by the processor, a camera to ground alignment value based on the ground normal vector; and generating, by the processor, second image data based on the camera to ground alignment value.

A multimedia system is provided for a scalable infotainment system of a motor vehicle. The scalable infotainment system includes a camera, which is releasably attached to the motor vehicle and supports camera software components for generating a media signal. The multimedia system includes a media player and a computer attached to the motor vehicle. The computer has a processor and a non-transitory computer readable storage medium, which stores player software components including multiple streaming protocols used by the media player to play a media stream. The processor is programmed to transmit to the camera a player setup signal including data associated with the player software components. The processor is further programmed to receive from the camera a media signal associated with a media stream generated by the camera by using a common one of the streaming protocols supported by the camera and the media player.

Multimedia system for a scalable infotainment system of a motor vehicle

Systems and methods are provided for generating alignment parameters for processing data associated with a vehicle. In one embodiment, a method includes: receiving image data associated with an environment of the vehicle; receiving lidar data associated with the environment of the vehicle; processing, by a processor, the image data to determine data points associated with at least one pole identified within image data; processing, by the processor, the lidar data to determine data points associated with at least one pole identified within the lidar data; selectively storing the data points as data point pairs in a data buffer; iteratively processing, by the processor, the data point pairs with a plurality of perturbations to determine a transformation matrix; generating, by the processor, alignment data based on the transformation matrix; and processing future data based on the alignment parameters.

Methods and systems for camera to lidar alignment using road poles

A scan aggregator and filter for an autonomous vehicle includes a plurality of radar sensors, where each radar sensor performs a plurality of individual scans of a surrounding environment to obtain data in the form of a radar point cloud including a plurality of detection points. The scan aggregator and filter also includes an automated driving controller in electronic communication with the plurality of radar sensors. The automated driving controller is instructed to filter each of the individual scans to define a spatial region of interest and to remove the detection points of the radar point cloud that represent moving objects based on a first outlier-robust model estimation algorithm. The automated driving controller aggregates a predefined number of individual scans together based on a motion compensated aggregation technique to create an aggregated data scan and applies a plurality of density-based clustering algorithms to filter the aggregated data scan.

Filtering and aggregating detection points of a radar point cloud for an autonomous vehicle

A calibration pipeline for 6DoF alignment parameters for an autonomous vehicle includes an automated driving controller instructed to receive inertial measurement unit (IMU) poses and final radar poses and determine smoothened IMU poses from the IMU poses and smoothened final radar poses from the final radar poses. The automated driving controller aligns the smoothened IMU poses and the smoothened final radar poses with one another to create a plurality of radar-IMU A, B relative pose pairs. The automated riving controller determines a solution yielding a threshold number of inliers of further filtered radar-IMU A, B relative pose pairs, randomly samples the further filtered radar-IMU A, B relative pose pairs with replacements several times to determine a stream of filtered radar-IMU A, B relative pose pairs, and solves for a solution X for the stream of filtered radar-IMU A, B relative pose pairs.

Calibration pipeline for estimating six degrees of freedom (6DoF) alignment parameters for an autonomous vehicle

Growing community of inventors

Troy, MI, United States of America

Wende Zhang