Anatomical segmentation identifying modes and viewpoints with deep learning across modalities

Abstract

A mechanism is provided in a data processing system comprising a processor and a memory, the memory comprising instructions that are executed by the processor to specifically configure the processor to implement a multi-modal classification and segmentation engine for anatomical segmentation identifying modes and viewpoints in biomedical images. The mechanism trains a neural network perform simultaneous classification and segmentation using a set of training images. The neural network provides a classification output that identifies a class label and a second output that identifies a segmentation label. The multi-modal classification and segmentation engine provides a biomedical image as the input image to the neural network. The neural network outputs a plurality of class label probabilities for a plurality of class labels and a plurality of segmentation label probabilities for each of a plurality of segmentation labels. A post-processing component executing within the multi-modal classification and segmentation engine classifies the biomedical image as an identified modality and an identified viewpoint based on the plurality of class label probabilities. The multi-modal classification and segmentation engine segments the biomedical image based on the plurality of segmentation label probabilities. The multi-modal classification and segmentation engine outputs the classified and segmented biomedical image.