Multimodal 3d object detection using temporal and structure consistency in voxel feature space

Abstract

An example device for detecting objects through processing of media data, such as image data and point cloud data, includes a processing system configured to form voxel representations of a real-world three-dimensional (3D) space using images and point clouds captured for the 3D space at consecutive time steps, extract image and/or point cloud features for voxels in voxel representations of the 3D space, determine correspondences between the voxels at consecutive time steps according to similarities between the extracted features, and determine positions of objects in the 3D space using the correspondences between the voxels. For example, the processing system may perform triangulation according to positions of a moving object to positions of the voxels at the time steps. In this manner, the processing system may generate an accurate bird's eye view (BEV) representation of the real-world 3D space.