Processing multimodal images of tissue for medical evaluation

Abstract

Methods and systems are provided for processing different-modality digital images of tissue. The method includes, for each image, detecting biological entities in the image and generating an entity graph comprising entity nodes, representing respective biological entities, interconnected by edges representing interactions between entities represented by the entity nodes. The method also includes selecting, from each image, anchor elements comprising elements corresponding to anchor elements of at least one other image, and generating an anchor graph in which anchor nodes, representing respective anchor elements, are interconnected with entity nodes of the entity graph for the image by edges indicating relations between entity nodes and anchor nodes. The method further includes generating a multimodal graph by interconnecting anchor nodes of the anchor graphs for different images via correspondence edges indicating correspondence between anchor nodes, and processing the multimodal graph to output multimodal data, derived from the plurality of images, for medical evaluation.