System and method for data adaptive single-shot multi-label segmentation with foundation models

Abstract

A method includes obtaining a medical image and receiving a selection of both a template image and a region of interest within the template image. The method includes inputting both the medical image and the template image into a trained vision transformer model and outputting from the trained vision transformer model both pixel level feature vectors from the medical image and a reference pixel level feature vector from the region of interest of the template image. The method includes inputting both the pixel level feature vectors and the reference pixel level feature vector into a trained contrastive similarity metric learning model and outputting from the trained contrastive similarity metric learning model pixel that are similar to reference pixels. The method includes labeling the pixels in the medical image with a segmentation mask, wherein the pixels that are labeled in the medical image correspond to the region of interest.