Multimodal data inference

Abstract

Computer-implemented methods are provided for generating machine learning model for multimodal data inference tasks. Such a method includes, for each sample in a training dataset of multimodal data samples, encoding the sample to produce a compressed vector representation of the sample in a k-dimensional latent space, and perturbing features of the sample to identify, for each dimension of the latent space, a set of active features perturbation of each of which produces more than a threshold change in the vector representation in that dimension. The method further comprises generating a sample graph having nodes interconnected by edges, wherein the nodes comprise nodes representing respective said features of the sample and edges interconnecting nodes indicate the active features for each dimension. The sample graph is then used to train a graph neural network model to perform the multimodal data inference task. Multimodal data inference systems employing such models are also provided.