Multimodal ai model protection using embeddings

Abstract

Techniques for assessing multi-modal inputs to a machine learning model involve receiving a multimodal input containing an image, producing several transformed versions of that image, and generating embeddings for both the original and transformed images. A pairwise similarity analysis among all embeddings is conducted to determine distance values. Two dissimilarity metrics can then be calculated: one reflecting the differences among the transformed images, and another comparing the original image to its transformed versions. If the dissimilarity among the transformed images is greater than that between the original and transformed images plus a threshold, the system triggers a remediation action. This action either blocks the input from being processed by the machine learning model or prevents the model's output from being returned to the requester, thereby enhancing the reliability and security of the model.