Multimodal self-supervised learning for image analysis

Abstract

A machine learning model may be trained on a first task of puzzle solving before being tuned on a second task of image analysis. The training of the machine learning model may be self-supervised whereas the tuning of the machine learning model may be supervised. The training data may include a puzzle generated to include multiple imaging modalities. The puzzle may be generated by shuffling a position of the pieces forming an original image. The machine learning model may be trained to perform the first task by reassembling the pieces in the puzzle to generate a reconstruction of the original image. Upon being trained to perform the first task and tuned to perform the second task, the machine learning model may be deployed to perform the second task. The second task may be an image segmentation task such as tumor segmentation and a regression task such as survival prediction.