Multi-modal fusion techniques considering inter-modality correlations and computer model uncertainty

Abstract

A joint multimodal fusion computer model architecture is provided that receives prediction output data from a machine learning (ML) computer model set comprising a plurality of different subsets of ML computer models operating on input data of different modalities and generating different prediction outputs. Prediction outputs are fused by executing an uncertainty and correlation weighted (UCW) joint multimodal fusion operation on the prediction outputs to generate a fused output providing multimodal prediction output data. The UCW joint multimodal fusion operation applies different weights to different ones of prediction outputs from the different subsets of ML computer models operating on input data of different modalities. The different weights are determined based on an estimation of uncertainty in each of the different subsets of ML computer models and an estimate of a correlation between different modalities.