Feature fusion for multi-modal machine learning analysis

Abstract

A system to perform multi-modal analysis has at least three distinct characteristics: an early abstraction layer for each data modality integrating homogeneous feature cues coming from different deep learning architectures for that data modality, a late abstraction layer for further integrating heterogeneous features extracted from different models or data modalities and output from the early abstraction layer, and a propagation-down strategy for joint network training in an end-to-end manner. The system is thus able to consider correlations among homogeneous features and correlations among heterogenous features at different levels of abstraction. The system further extracts and fuses discriminative information contained in these models and modalities for high performance emotion recognition.