Optimizations for real-time sensor fusion in vehicle understanding models

Abstract

Autonomous vehicles utilize perception and understanding of vehicles to predict behaviors of the vehicles, and to plan a trajectory. Understanding of attributes of vehicles may be improved through sensor fusion. Sensor fusion can be computationally expensive and may be difficult to implement in a real-time vehicle understanding system. To limit computational complexity while benefiting from machine learning across modalities, sensor fusion may be selectively implemented for a subset of task groups of a multi-task machine learning model. In some cases, part-based understanding may be implemented before fusion to limit the features being fused together to part features that are most salient for the task group. In addition, sensor data and features that may be fused together can be limited to sensor data and features within a desired field of view. A model that implements sensor fusion may be disabled for objects that are beyond a threshold distance.