Definiens Ag

Astrazeneca Computational Pathology Gmbh

Nicolas Brieu

Guenter Schmidt

Ansh Kapil

Olivier Pauly

Jan Martin Lesniak

Johannes Zimmermann

Tobias Wiestler

Simon Lanzmich

Moritz Widmaier

Ralf Schoenmeyer

Mehmet Yigitsoy

Melanie Goerner

Günter Schmidt

Melanie Görner

A convolutional neural network predicts which regions of a tissue slice would be stained by a first stain by training a model to identify those regions based only on tissue stained by a second stain. Thereafter the first stain need not be used to mark cancerous regions on other tissue slices that are stained with the second stain. The training slice is stained with a first immunohistochemical stain and a second counterstain. A target region of an image of the training slice is identified using image analysis based on the first stain. A set of parameters for associated mathematical operations are optimized to train the model to classify pixels of the image as belonging to the target region based on the second stain but not on the first stain. The trained parameters are stored in a database and applied to other images of tissue not stained with the first stain.

Using a first stain to train a model to predict the region stained by a second stain

A method for indicating how a cancer patient will respond to a predetermined therapy relies on spatial statistical analysis of classes of cell centers in a digital image of tissue of the cancer patient. The cell centers are detected in the image of stained tissue of the cancer patient. For each cell center, an image patch that includes the cell center is extracted from the image. A feature vector is generated based on each image patch using a convolutional neural network. A class is assigned to each cell center based on the feature vector associated with each cell center. A score is computed for the image of tissue by performing spatial statistical analysis based on classes of the cell centers. The score indicates how the cancer patient will respond to the predetermined therapy. The predetermined therapy is recommended to the patient if the score is larger than a predetermined threshold.

Deep learning method for predicting patient response to a therapy

A score of a histopathological diagnosis of cancer is generated by loading an image patch of an image into a processing unit, determining how many pixels of the image patch belong to a first tissue, processing additional image patches cropped from the image to determine how many pixels of each image patch belong to the first tissue, computing the score and displaying it along with the image on a graphical user interface. The image patch is cropped from the image of a slice of tissue that has been immunohistochemically stained using a diagnostic antibody. The first tissue comprises tumor epithelial cells that are positively stained by the diagnostic antibody. Determining how many pixels belong to the first tissue is performed by processing the image patch using a convolutional neural network. The score of the histopathological diagnosis is computed based on the total number of pixels belonging to the first tissue.

Deep learning method for tumor cell scoring on cancer biopsies

Both pixel-oriented analysis and the more accurate yet slower object-oriented analysis are used to recognize patterns in images of stained cancer tissue. Images of tissue from other patients that are similar to tissue of a target patient are identified using the standard deviation of color in the images. Object-oriented segmentation is then used to segment small portions of the images of the other patients into object exhibiting object characteristics. Pixelwise descriptors associate each pixel in the remainder of the images with object characteristics based on the color of pixels at predetermined offsets from the characterized pixel. Pixels in the image of the target patient are assigned object characteristics without performing the slow segmentation of the image into objects. A pixel heat map is generated from the target image by assigning pixels the color corresponding to the object characteristic that the pixelwise descriptors indicate is most likely associated with each pixel.

Applying pixelwise descriptors to a target image that are generated by segmenting objects in other images

Both object-oriented analysis and the faster pixel-oriented analysis are used to recognize patterns in an image of stained tissue. Object-oriented image analysis is used to segment a small portion of the image into object classes. Then the object class to which each pixel in the remainder of the image most probably belongs is determined using decision trees with pixelwise descriptors. The pixels in the remaining image are assigned object classes without segmenting the remainder of the image into objects. After the small portion is segmented into object classes, characteristics of object classes are determined. The pixelwise descriptors describe which pixels are associated with particular object classes by matching the characteristics of object classes to the comparison between pixels at predetermined offsets. A pixel heat map is generated by giving each pixel the color assigned to the object class that the pixelwise descriptors indicate is most probably associated with that pixel.

Learning pixel visual context from object characteristics to generate rich semantic images

The coregistration of digital images of tissue slices is improved by updating landmarks based on the manual outlining of regions of interest on the images. A first image of a first slice is coarsely coregistered with a second image of a second slice using a first landmark on the first image and a second landmark on the second image. A user manually outlines a first region of interest on the first image. The outline is positioned over a second region of interest on the second image using the second landmark. The user manually moves a contour point of the outline on the second image to form a corrected outline. The second landmark is moved based on how the contour point was manually moved so that the first and second images are more finely coregistered after the second landmark is moved. Each state of corrected contour points and landmarks is saved.

Updating landmarks to improve coregistration as regions of interest are corrected

A method for coregistering images involves defining middle paths through image objects depicting tissue slices of needle biopsies. First landmarks are defined on a first middle path through a first image object in a first digital image of a first tissue slice, and second landmarks are defined on a second middle path through a second image object of a second digital image of a second tissue slice. Individual first landmarks are associated with individual second landmarks. A first pixel in the first object is coregistered with a second pixel in the second object using multiple first and second landmarks. The first image is displayed in a first frame on a graphical user interface, and the second image is displayed in a second frame such that the first pixel is centered in the first frame, the second pixel is centered in the second frame, and the images have the same orientations.

Growing community of inventors

Munich, Germany

Nicolas Brieu