Machine learning on multimodal chemical and whole slide imaging data for predicting cancer labels directly from tissue images

Abstract

Systems, methods and programs including machine learning techniques which can predict a spectral image directly from an optical image of a patient's tissue sample using a first model. There may be different first models for different cancers. Each first model may be trained by using a plurality of pairs of images from different samples, respectively, where each image in a respective pair may be obtained via different imaging modalities. The systems, methods and programs may also include machine learning techniques which can predict cancer labels from the predicted spectral image using a second model. There may be different second models for different cancers. Each second model may be trained using multiple spectral images and corresponding manually input cancer labels.