Tsinghua University

Shiyan Zhang

Li Wang

Lijun Zhao

Jun Li

Lei Zhu

Zhiwei Li

Lei Yang

Xinyu Zhang

Tianlei Zhang

Xingang Wu

Hanwen Gao

A vision-LiDAR fusion method and system based on deep canonical correlation analysis are provided. The method comprises: collecting RGB images and point cloud data of a road surface synchronously; extracting features of the RGB images to obtain RGB features; performing coordinate system conversion and rasterization on the point cloud data in turn, and then extracting features to obtain point cloud features; inputting point cloud features and RGB features into a pre-established and well-trained fusion model at the same time, to output feature-enhanced fused point cloud features, wherein the fusion model fuses RGB features to point cloud features by using correlation analysis and in combination with a deep neural network; and inputting the fused point cloud features into a pre-established object detection network to achieve object detection. A similarity calculation matrix is utilized to fuse two different modal features.

Vision-LiDAR fusion method and system based on deep canonical correlation analysis

Growing community of inventors

Beijing, China

Shiyan Zhang