Object detection using transformer based fusion of multi-modality sensor data

Abstract

Disclosed are techniques for combining data using transformer-based machine learning models. In some examples, a first transformer is used to combine a first dataset with a second dataset. The results are then combined with a third dataset, using a second transformer. Each dataset can represent data from a different sensor modality. The transformers compute scores based on queries and apply the scores to values. The first dataset can be used to generate queries for the first transformer, and the values for the first transformer can be derived from the second dataset. Similarly, the third dataset can be used to generate queries for the second transformer, and the values for the second transformer can be derived from the output of the first transformer. The output of the second transformer is therefore a combination of all three datasets and can be used for object detection, for example, determining three-dimensional boundaries of objects.