International Business Machines Corporation

University of Southern California

Hongzhi Wang

Tanveer Fathima Syeda-Mahmood

Mehdi Moradi

Rui Zhang

Yaniv Gur

Alexandros Karargyris

Joy Tzung-yu Wu

Gopalkrishna Veni

Ehsan Dehghan Marvast

Chun Lok Wong

Deepika Kakrania

John Paul Francis

Yu Long Cao

Renu Tewari

Lukas Rupprecht

Ahmed El Harouni

Subashini Balachandran

Anup Pillai

Ashutosh Jadhav

Satyananda Kashyap

Hui Tang

Prasanth Prasanna

Vaishnavi Subramanian

A method and system for automatically inferring a subject's body position in a two-dimensional image produced by a medical-imaging system are disclosed. The image is labeled with a body position selected from a semantically meaningful set of candidate positions sequenced in order of their relative locations in a subject's body. A processor performs procedures that each identify a class of image features related to pixel intensity, such as a histogram of gradients, local binary patterns, or Haar-like features. A second set of procedures employs applications of a pretrained convolutional neural network that has learned to recognize features of a specific class of medical images. The results of both types of procedures are then mapped by a pretrained support-vector machine onto candidate image labels, which are mathematically combined into a single, semantically meaningful, label most likely to identify a body position of the subject shown by the image.

Identifying subject matter of a digital image

A joint multimodal fusion computer model architecture is provided that receives prediction output data from a machine learning (ML) computer model set comprising a plurality of different subsets of ML computer models operating on input data of different modalities and generating different prediction outputs. Prediction outputs are fused by executing an uncertainty and correlation weighted (UCW) joint multimodal fusion operation on the prediction outputs to generate a fused output providing multimodal prediction output data. The UCW joint multimodal fusion operation applies different weights to different ones of prediction outputs from the different subsets of ML computer models operating on input data of different modalities. The different weights are determined based on an estimation of uncertainty in each of the different subsets of ML computer models and an estimate of a correlation between different modalities.

Multi-modal fusion techniques considering inter-modality correlations and computer model uncertainty

A method for automatically training and applying automatic segmentation in digital image processing is provided. The method may include, in response to receiving a plurality of digital images wherein each digital image associated with the plurality of digital images comprises only one annotated structure out of a plurality of structures included in each digital image, applying a predictive algorithm to each digital image that determines a predicted probability of each annotation in each digital image, determines a predicted background for each digital image, and merges the predicted probability of each annotation with the predicted background in each digital image. The method may further include, in response to applying the predictive algorithm, using the received plurality of digital images to train and apply an application for automatically segmenting unlabeled digital images.

Reducing structural redundancy in automatic image segmentation

Methods and systems for training a model labeling two or more organic structures within an image. One method includes receiving a set of training images. The set of training images including a first plurality of images and a second plurality of images. Each of the first plurality of images including a label for each of the two or more organic structures and each of the second plurality of images including a label for only a subset of the two or more organic structures. The method further includes training the model using the first plurality of images, the second plurality of images, and a label merging function mapping a label from the first plurality of images to a label included in the second plurality of images.

Model training using fully and partially-annotated images

Methods and systems for training a model labeling two or more organic structures in an image. One method includes receiving a set of training images including a first plurality of images and a second plurality of images. Each of the first plurality of images including a label for a first subset of the two or more organic structures and each of the second plurality of images including a label for a second subset of the two or more organic structures, the second subset being different than the first subset. The method also includes training the model using the first plurality of images, the second plurality of images, and a label merging function mapping a label included in the first plurality of images to a label included in the second plurality of images.

Model training using partially-annotated images

Embodiments of the present disclosure include a computer-implemented method, a system, and a computer program product for integrating bottom-up segmentation techniques into a semi-supervised image segmentation machine learning model. The computer implemented method includes training a machine learning model with a labeled dataset. The labeled dataset includes ground truth segmentation labels for each sample in the labeled dataset. The computer implemented method also includes generating a pseudo labeled dataset by applying an unlabeled dataset to the machine learning model using a top-down segmentation grouping rule. The computer implemented method further includes evaluating the pseudo labeled dataset using a bottom-up segmentation grouping rule to produce evaluation results, combining the pseudo labeled dataset with the second pseudo labeled dataset into a training dataset, and then retraining the machine learning model with the training dataset.

Integrated bottom-up segmentation for semi-supervised image segmentation

A system for automated detection and type classification of central venous catheters. The system includes an electronic processor that is configured to, based on an image, generate a segmentation of a potential central venous catheter using a segmentation method and extract, from the segmentation, one or more image features associated with the potential central venous catheter. The electronic processor is also configured to, based on the one or more image features, determine, using a first classifier, whether the image includes a central venous catheters and determine, using a second classifier, a type of central venous catheter included in the image.

Automated detection and type classification of central venous catheters

Mechanisms are provided to implement an automated medical imaging report generator which receives an input medical image and inputs the input medical image into a machine learning (ML) computer model trained to predict finding labels based on patterns of image features extracted from the medical image. The ML computer model generates a prediction of a finding label applicable to the input medical image in terms of a finding label prediction output vector. Based on the finding label prediction output vector, a lookup operation is performed, in a medical report database of previously processed medical imaging report data structures, to find a matching medical imaging report data structure corresponding to the finding label. An output medical imaging report is generated for the input medical image based on natural language content of the matching medical imaging report data structure.

Automatic generation of medical imaging reports based on fine grained finding labels

Example-based affine registration is provided. In various embodiments, a plurality of training images is read. A predetermined affine transform is read for each of the plurality of training images. Each affine transform maps its associated image to a template. Weights are determined for each of the plurality of training images. The weights are determined to minimize a difference between the test image and a weighted linear combination of the training images. An affine transform is determined mapping the test image to the template by computing a weighted linear combination of the affine transforms using the weights.

Techniques for example-based affine registration

A mechanism is provided in a data processing system comprising a processor and a memory, the memory comprising instructions executed by the processor to specifically configure the processor to implement a multi-atlas segmentation engine. An offline registration component performs registration of a plurality of atlases with a set of image templates to thereby generate and store, in a first registration storage device, a plurality of offline registrations. The atlases are annotated training medical images and the image templates are non-annotated medical images. The multi-atlas segmentation engine receives a target image. An image selection component selects a subset of image templates in the set of image templates based on the target image. An online registration component performs registration of the subset of image templates with the target image to generate a plurality of online registrations. The multi-atlas segmentation engine retrieves offline registrations corresponding to the subset of image templates from the first registration storage device. The multi-atlas segmentation engine performs segmentation of the target image based on the retrieved offline registrations corresponding to the subset of image templates and the plurality of online registrations. The segmentation applies labels to anatomical structures present in the target image based on the retrieved offline registrations and the plurality of online registrations to thereby output a modified target image.

Template based anatomical segmentation of medical images

Assigning a semantically meaningful label to a digital image

Various embodiments are provided for managing memory for image analysis in a computing environment by a processor. An estimated amount of memory may be reserved for an image analysis job. During a processing of the image analysis job, at least one thread of the image analysis job is partially suspended by restricting the image analysis job from requesting allocation of additional memory upon memory requirements for the image analysis job exceeding the estimated amount of memory. Commensurate with partially suspending the at least one thread, a state of the image analysis job is maintained in the memory notwithstanding the processing of the at least one thread associated with the state is suspended.

Memory management for complex image analysis

A method comprises (a) collecting (i) a set of chest computed tomography angiography (CTA) images scanned in the axial view and (ii) a manual segmentation of the images, for each one of multiple organs; (b) preprocessing the images such that they share the same field of view (FOV); (c) using both the images and their manual segmentation to train a supervised deep learning segmentation network, wherein loss is determined from a multi-dice score that is the summation of the dice scores for all the multiple organs, each dice score being computed as the similarity between the manual segmentation and the output of the network for one of the organs; (d) testing a given (input) pre-processed image on the trained network, thereby obtaining segmented output of the given image; and (e) smoothing the segmented output of the given image.

System for segmentation of anatomical structures in cardiac CTA using fully convolutional neural networks

One embodiment provides a method comprising receiving image data with a first image resolution, and determining an optimal image resolution for sampling the image data based on a learned model. The optimal image resolution is lower than the first image resolution. The method further comprises sampling the image data at the optimal image resolution, and performing image analysis on sampled image data resulting from the sampling.

Growing community of inventors

San Jose, CA, United States of America

Hongzhi Wang