Systems and methods for multimodal fusion of missing and unpaired image and tabular data for defect classification

Abstract

A method may include providing a data set including rows of data. The rows of data may include at least one row of unpaired modality including a first modality, and at least one row of paired modality may include both the first modality and a second modality. The method may further include imputing, by a modality-specific encoder, the at least one row of unpaired modality by interpolating embeddings from the second modality of the paired modality; training, in a latent space, the modality-specific encoder based on the imputation for unimodal prediction and bimodal prediction; and generating a confidence value for the unimodal prediction and the bimodal prediction.