System and method of training vision transformer on small-scale datasets

Abstract

A deep learning training system and method, includes an imaging system for capturing medical images, a machine learning engine, and display. The machine learning engine selects a small-scale of images from a training dataset, generates global views by randomly selecting regions in one image, generates local views by randomly selecting regions covering less than a majority of the image, receives the generated global views as a first sequence of non-overlapping image patches, receives the generated global views and the generated local views as a second sequence of non-overlapping image patches, trains parameters in a student-teacher network to predict a class of objects by self-supervised view prediction using the first sequence and the second sequence. The teacher parameters are updated via exponential moving average of the student network parameters. The parameters in the teacher network are transferred to the vision transformer, and the vision transformer is trained by supervised learning.