Gm Global Technology Operations LLC

Carnegie Mellon University

Wende Zhang

Jinsong Wang

Bakhtiar Brian Litkouhi

Hao Yu

Kent S Lybecker

Xinyu Du

Sanjeev Manubhai Naik

Yao Hu

Ryan M Frakes

Shuqing Zeng

Fan Bai

Jeffrey S Piasecki

Qi Wu

Donald K Grimm

Esther Anderson

Qingrong Zhao

Binbin Li

Tsuhan Chen

Travis S Hester

Varsha K Sadekar

Yousef A Omar

Mark J Wolski

Xiujie Gao

Mohannad F Murad

Wei Tong

Dan Levi

Kamran Ali

Fernando De La Torre

Mark Edward Shields

Debbie E Nachtegall

Mark T Gaylord

Timothy J Talty

Robert Alan Dziurda

Vijayakumar Bhagavatula

Qi Zhang

Caroline Chung

Xi Ju

Jiang L Du

Zhencheng Hu

Adrienne M Williamson

Timothy J Canner

Nancy L Johnson

Thomas A Seder

Christopher Paul Urmson

Paul Edward Krajewski

Xiaosong Huang

Upali Priyantha Mudalige

Charles A Green

Norman Joseph Weigert

Jin-Woo Lee

Duane S Carper

Allan K Lewis

Sai Vishnu Aluru

Xin Yu

Joseph G Machak

Robert D Sims, Iii

Chaminda Basnayake

John Sergakis

Chuan Li

Michael A Wuergler

Jeffrey A Bozeman

Cem U Saraydar

Michael T Chaney, Jr

Mohsen Khalili

Jace C Stokes

Brunno L Moretti

Christopher A Stanek

Kevin K Hoang

Dong Huang

Xuehan Xiong

Veronique C Prinet

Franscisco Vicente

Kevin Peterson

Paul Andrew Justin

Hongtao Wang

Alma Bushi

Gary V Bandurski

Xinhua Xiao

Jie Yin

Francisco Vicente

Zehua Huang

James Clem

Vernon Hole

A sensor alignment method including detecting, detecting an object within a first sensor field of view and a second sensor field of view, detecting a plurality of depth point clouds of the first field of view in response to the object being stationary, the host vehicle being stationary, and the distance between the object and the host vehicle being less than a threshold distance, aggregating the plurality of depth point clouds into an aggregated depth point cloud, detecting a first location of an edge of the object in response to the aggregated depth point cloud, detecting a second location of the edge of the object using an edge detection algorithm on the image, generating a lidar to camera alignment in response to a difference between the first location and the second location.

Robust lidar-to-camera sensor alignment

Methods, apparatus, and systems for sensor alignment include acquiring a translational vector, a first calibration point location and a second calibration point location, determining an expected rotational orientation in response to the translational vector, the first calibration point location and the second calibration point location, capturing an image of the a first calibration point and the second calibration point, determining a first position of the first calibration point and a second position of the second calibration point in response to the image, calculating a calculated rotational orientation in response to the first position of the first calibration point and the second position of the second calibration point, determining a calibration value in response to the calculated rotational orientation, storing the calibration value and controlling a vehicle in response to the calibration value and a subsequent image.

Sensor angular alignment

A system in a vehicle includes a lidar system to obtain lidar data in a lidar coordinate system, and processing circuitry to obtain the lidar data and localization data. The localization data indicates a location and orientation of the vehicle. The processing circuitry automatically determines an alignment state resulting in a lidar-to-vehicle transformation matrix that projects the lidar data from the lidar coordinate system to a vehicle coordinate system to provide lidar-to-vehicle data. The alignment state is determined using the localization data.

Automatic detection of lidar to vehicle alignment state using localization data

A system includes a platform operational to move through an environment, a navigation system operational to measure platform poses of the platform in the environment, a camera operational to generate images of the environment at multiple timestamps, and a computer. The computer is operational to receive a first image from the camera, receive a second image from the camera, estimate a view direction of the first camera pose based on a first and second platform poses and alignment parameters, generate a rendered portion of the environment with a neural radiance field technique based on the second image and the alignment parameters, generate a predicted image of the environment as observed along the view direction through the rendered portion of the environment, determine differences between the first image and the predicted image, and update one or more of the plurality of alignment parameters based on the one or more differences.

Neural radiance field based camera alignment

A system in a vehicle includes a lidar system to transmit incident light and receive reflections from one or more objects as a point cloud of points. The system also includes processing circuitry to identify feature points among the points of the point cloud, the feature points being horizontal feature points reflected from a horizontal surface or vertical feature points reflected from a vertical surface. The processing circuitry processes the point cloud by obtaining a normal vector corresponding to each of the points of the point cloud. The normal vector includes a first component associated with a first dimension, a second component associated with a second dimension, and a third component associated with a third dimension.

Identification of planar points in lidar point cloud obtained with vehicle lidar system

A system in a vehicle includes a lidar system to transmit incident light and receive reflections from one or more objects as a point cloud of points. The system also includes processing circuitry to identify planar points and to identify edge points of the point cloud. Each set of planar points forms a linear pattern and each edge point is between two sets of planar points, and the processing circuitry identifies each point of the points of the point cloud as being within a virtual beam among a set of virtual beams. Each virtual beam of the set of virtual beams representing a horizontal band of the point cloud.

Virtual beams for identification of edge and planar points in lidar point cloud obtained with vehicle lidar system

A LIDAR-to-LIDAR alignment system includes a memory and an autonomous driving module. The memory stores first and second points based on outputs of first and second LIDAR sensors. The autonomous driving module performs a validation process to determine whether alignment of the LIDAR sensors satisfy an alignment condition. The validation process includes: aggregating the first and second points in a vehicle coordinate system to provide aggregated LIDAR points; based on the aggregated LIDAR points, performing (i) a first method including determining pitch and roll differences between the first and second LIDAR sensors, (ii) a second method including determining a yaw difference between the first and second LIDAR sensors, or (iii) point cloud registration to determine rotation and translation differences between the first and second LIDAR sensors; and based on results of the first method, the second method or the point cloud registration, determining whether the alignment condition is satisfied.

Online validation of LIDAR-to-LIDAR alignment and LIDAR-to-vehicle alignment

A LIDAR-to-vehicle alignment system includes a memory and alignment and autonomous driving modules. The memory stores points of data provided based on an output of one or more LIDAR sensors and localization data. The alignment module performs an alignment process including: based on the localization data; determining whether a host vehicle is turning; in response to the host vehicle turning; selecting a portion of the points of data; aggregating the selected portion to provide aggregated data; selecting targets based on the aggregated data; and based on the selected targets, iteratively reducing a loss value of a loss function to provide a resultant LIDAR-to-vehicle transformation matrix. The autonomous driving module: based on the resultant LIDAR-to-vehicle transformation matrix, converts at least the selected portion to at least one of vehicle coordinates or world coordinates to provide resultant data; and performs one or more autonomous driving operations based on the resultant data.

Aggregation-based LIDAR data alignment

A system for on-vehicle camera alignment monitoring includes an on-vehicle camera in communication with a controller. The controller monitors vehicle operating parameters and camera signal parameters, and captures an image file from the on-vehicle camera. A first level analysis of the image file, the vehicle operating parameters, and the camera signal parameters is executed to detect dynamic conditions and image feature parameters that affect camera alignment. An error with one of the dynamic conditions or the image feature parameters that affects the camera alignment is detected. A second level analysis of the camera signal parameters is executed to identify a root cause indicating one of the dynamic conditions or the image feature parameters that affects the camera alignment based upon the error. A camera alignment-related fault is detected based upon the root cause, and vehicle operation is controlled based upon the camera alignment-related fault.

On-vehicle camera alignment monitoring system

Systems and methods for a vehicle are provided. In one embodiment, a method includes: receiving image data defining a plurality of images associated with an environment of the vehicle; determining, by a processor, feature points within at least one image of the plurality of images; selecting, by the processor, a subset of the feature points as ground points based on a fixed two dimensional image road mask and a three dimensional region; determining, by the processor, a ground plane based on the subset of feature points; determining, by the processor, a ground normal vector from the ground plane; determining, by the processor, a camera to ground alignment value based on the ground normal vector; and generating, by the processor, second image data based on the camera to ground alignment value.

Methods and systems for camera to ground alignment

Methods and systems for a vehicle are provided. In one embodiment, the method includes: receiving image data defining a plurality of images associated with an environment of the vehicle; determining, by a processor, feature points within at least one image of the plurality of images; selecting, by the processor, a subset of the feature points as ground points; determining, by the processor, a ground plane based on the subset of feature points; determining, by the processor, a ground normal vector from the ground plane; determining, by the processor, the ground normal vector based on a sliding widow method; determining, by the processor, a camera to ground alignment value based on the ground normal vector; and generating, by the processor, second image data based on the camera to ground alignment value.

A vehicle control system including a spatial monitoring system includes on-vehicle cameras that capture images, from which are recovered a plurality of three-dimensional points. A left ground plane normal vector is determined for a left image, a center ground plane normal vector is determined for a front image, and a right ground plane normal vector is determined for a right image. A first angle difference between the left ground plane normal vector and the center ground plane normal vector is determined, and a second angle difference between the right ground plane normal vector and the center ground plane normal vector is determined. An uneven ground surface is determined based upon one of the first angle difference or the second angle difference, and an alignment compensation factor for the left camera or the right camera is determined. A bird's eye view image is determined based upon the alignment compensation factor.

On-vehicle spatial monitoring system

A scan matching and radar pose estimator for determining a final radar pose for an autonomous vehicle includes an automated driving controller that is instructed to determine a hyper-local submap based on a predefined number of consecutive aggregated filtered data point cloud scans and associated pose estimates. The automated driving controller determines an initial estimated pose by aligning a latest aggregated filtered data point cloud scan with the most recent hyper-local submap based on an iterative closest point (ICP) alignment algorithm. The automated driving controller determines a pose graph based on the most recent hyper-local submap and neighboring radar point cloud scans, and executes a multi-view non-linear ICP algorithm to adjust initial estimated poses corresponding to the neighboring radar point cloud scans in a moving window fashion to determine a locally adjusted pose.

Scan matching and radar pose estimator for an autonomous vehicle based on hyper-local submaps

Systems and methods for a vehicle are provided. In one embodiment, a method includes: receiving image data defining a plurality of images associated with an environment of the vehicle; receiving vehicle data indicating a velocity of the vehicle, wherein the vehicle data is associated with the image data; determining, by a processor, feature points within at least one image based on the vehicle data and a three dimensional projection method; selecting, by the processor, a subset of the feature points as ground points; determining, by the processor, a ground plane based on the subset of feature points; determining, by the processor, a ground normal vector from the ground plane; determining, by the processor, a camera to ground alignment value based on the ground normal vector; and generating, by the processor, second image data based on the camera to ground alignment value.

A multimedia system is provided for a scalable infotainment system of a motor vehicle. The scalable infotainment system includes a camera, which is releasably attached to the motor vehicle and supports camera software components for generating a media signal. The multimedia system includes a media player and a computer attached to the motor vehicle. The computer has a processor and a non-transitory computer readable storage medium, which stores player software components including multiple streaming protocols used by the media player to play a media stream. The processor is programmed to transmit to the camera a player setup signal including data associated with the player software components. The processor is further programmed to receive from the camera a media signal associated with a media stream generated by the camera by using a common one of the streaming protocols supported by the camera and the media player.

Multimedia system for a scalable infotainment system of a motor vehicle

Growing community of inventors

Troy, MI, United States of America

Wende Zhang