Google Inc.

Tempus Labs, Inc.

Verily Life Sciences LLC

Martin Christian Stumpe

Lily Peng

Qian Yu

Nike Tsiapera Beaubier

Liron Yatziv

Yun Liu

Timo Kohlberger

Vinay Damodar Shet

Ashraf Hafez

Caroline Epstein

Sacha Christophe Arnoud

Daniel Neems

Adrian William George Lange

Dar-Shyang Lee

Marc Adlai Coram

Christian Szegedy

Oded Fuhrmann

Philip Charles Nelson

Dale R Webster

Lily Hao Yi Peng

Philip H Nelson

Ryan Poplin

Avinash Vaidyanathan Varadarajan

Arunachalam Narayanaswamy

Jason Hipp

Varun Gulshan

Derek Janme Wu

Katharine Blumer

Aly Azeem Khan

Dumitru Erhan

Krishna Kumar Gadepalli

Joshua S K Bell

Catherine Igartua

Matthew Symonds

Daniel Smilkov

Raphael A Pelossof

Varun Godbole

Kunal Nagpal

Davis Foote

Yeqing Li

Jackson Michuda

Kyle Ashley Beauchamp

Joshuah Kapilivsky

Calvin McCarter

Timothy Taxter

Sacha Christopher Arnoud

Aïcha Bentaieb

David F Steiner

Ellery Wulczyn

Naren Manoj

Po-Hsuan Cameron Chen

Craig Mermel

One example method for AI prediction of prostate cancer outcomes involves receiving an image of prostate tissue; assigning Gleason pattern values to one or more regions within the image using an artificial intelligence Gleason grading model, the model trained to identify Gleason patterns on a patch-by-patch basis in a prostate tissue image; determining relative areal proportions of the Gleason patterns within the image; assigning at least one of a risk score or risk group value to the image based on the determined relative areal proportions; and outputting at least one of the risk score or the risk group value.

Artificial intelligence prediction of prostate cancer outcomes

A machine learning predictor model is trained to generate a prediction of the appearance of a tissue sample stained with a special stain such as an IHC stain from an input image that is either unstained or stained with H&E. Training data takes the form of thousands of pairs of precisely aligned images, one of which is an image of a tissue specimen stained with H&E or unstained, and the other of which is an image of the tissue specimen stained with the special stain. The model can be trained to predict special stain images for a multitude of different tissue types and special stain types, in use, an input image, e.g., an H&E image of a given tissue specimen at a particular magnification level is provided to the model and the model generates a prediction of the appearance of the tissue specimen as if it were stained with the special stain. The predicted image is provided to a user and displayed, e.g., on a pathology workstation.

Virtual staining for tissue slide images

An augmented reality (AR) subsystem including one or more machine learning models, automatically overlays an augmented reality image, e.g., a border or outline, that identifies cells of potential interest, in the field of view of the specimen as seen through the eyepiece of an LCM microscope. The operator does not have to manually identify the cells of interest for subsequent LCM, e.g, on a workstation monitor, as in the prior art. The operator is provided with a switch, operator interface tool or other mechanism to select the identification of the cells, that is, indicate approval of the identification of the cells, while they view the specimen through the eyepiece. Activation of the switch or other mechanism invokes laser excising and capture of the cells of interest via a known and conventional LCM subsystem.

Augmented reality laser capture microdissection machine

A method for training a pattern recognizer to identify regions of interest in unstained images of tissue samples is provided. Pairs of images of tissue samples are obtained, each pair including an unstained image of a given tissue sample and a stained image of the given tissue sample. An annotation (e.g., drawing operation) is then performed on the stained image to indicate a region of interest. The annotation information, in the form of a mask surrounding the region of interest, is then applied to the corresponding unstained image. The unstained image and mask are then supplied to train a pattern recognizer. The trained pattern recognizer can then be used to identify regions of interest within novel unstained images.

Pathology predictions on unstained tissue

Systems and methods are provided for predicting metastasis of a cancer in a subject. A plurality of data elements for the subject's cancer is obtained, including sequence features comprising relative abundance values for gene expression in a cancer biopsy of the subject, optional personal characteristics about the subject, and optional clinical features related to the stage, histopathological grade, diagnosis, symptom, comorbidity, and/or treatment of the cancer in the subject, and/or a temporal element associated therewith. One or more models are applied to the plurality of data elements, determining one or more indications of whether the cancer will metastasize. A clinical report comprising the one or more indications is generated.

Predicting likelihood and site of metastasis from patient records

A method for generating a ground truth mask for a microscope slide having a tissue specimen placed thereon includes a step of staining the tissue specimen with hematoxylin and eosin (H&E) staining agents. A first magnified image of the H&E stained tissue specimen is obtained, e.g., with a whole slide scanner. The H&E staining agents are then washed from the tissue specimen. A second, different stain is applied to the tissue specimen, e.g., a special stain such as an IHC stain. A second magnified image of the tissue specimen stained with the second, different stain is obtained. The first and second magnified images are then registered to each other. An annotation (e.g., drawing operation) is then performed on either the first or the second magnified images so as to form a ground truth mask, the ground truth mask in the form of closed polygon region enclosing tumor cells present in either the first or second magnified image.

Method for creating histopathological ground truth masks using slide restaining

A system and method are provided for training and using a machine learning model to analyze hematoxylin and eosin (H&E) slide images, where the machine learning model is trained using a training data set comprising a plurality of unmarked H&E images and a plurality of marked H&E images, each marked H&E image being associated with one unmarked H&E image and each marked H&E image including a location of one or more molecules determined by analyzing a multiplex IHC image having at least two IHC stains, each IHC stain having a unique color and a unique target molecule. Predicted molecules and locations identified with the machine learning model result in an immunotherapy response class being assigned to the H&E slide image.

Systems and methods for generating histology image training datasets for machine learning models

A method is described for generating a prediction of a disease classification error for a magnified, digital microscope slide image of a tissue sample. The image is composed of a multitude of patches or tiles of pixel image data. An out-of-focus degree per patch is computed using a machine learning out-of-focus classifier. Data representing expected disease classifier error statistics of a machine learning disease classifier for a plurality of out-of-focus degrees is retrieved. A mapping of the expected disease classifier error statistics to each of the patches of the digital microscope slide image based on the computed out-of-focus degree per patch is computed, thereby generating a disease classifier error prediction for each of the patches. The disease classifier error predictions thus generated are aggregated over all of the patches.

Focus-weighted, machine learning disease classifier error prediction for microscope slide images

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for processing fundus images using fundus image processing machine learning models. One of the methods includes obtaining a model input comprising one or more fundus images, each fundus image being an image of a fundus of an eye of a patient; processing the model input using a fundus image processing machine learning model, wherein the fundus image processing machine learning model is configured to process the model input comprising the one or more fundus image to generate a model output; and processing the model output to generate health analysis data.

Processing fundus images using machine learning models

A microscope of the type used by a pathologist to view slides containing biological samples such as tissue or blood is provided with the projection of enhancements to the field of view, such as a heatmap, border, or annotations, substantially in real time as the slide is moved to new locations or changes in magnification or focus occur. The enhancements assist the pathologist in characterizing or classifying the sample, such as being positive for the presence of cancer cells or pathogens.

Augmented reality microscope for pathology

Systems and methods are provided for identifying a diagnosis of a cancer condition for a somatic tumor specimen of a subject. The method receives sequencing information comprising analysis of a plurality of nucleic acids derived from the somatic tumor specimen. The method identifies a plurality of features from the sequencing information, including two or more of RNA, DNA, RNA splicing, viral, and copy number features. The method provides a first subset of features and a second subset of features from the identified plurality of features as inputs to a first classifier and a second classifier, respectively. The method generates, from two or more classifiers, two or more predictions of cancer condition based at least in part on the identified plurality of features. The method combines, at a final classifier, the two or more predictions to identify the diagnosis of the cancer condition for the somatic tumor specimen of the subject.

Systems and methods for multi-label cancer classification

A method for aligning two different magnified images of the same subject includes a first step of precomputing rigid transformations for the two different images globally (i.e., for all or approximately all regions of the images). Pairs of corresponding features in the two different magnified images are identified and transformation values are assigned to each of the pairs of corresponding features. In a second step, while images are being generated for display to a user, a locally optimal rigid transformation for the current field of view is computed. In a third step the images are aligned as per the locally optimal rigid transformation. Non-zero weight is given to transformation values for pairs of features that are outside the current field of view. Typically, the second and third steps are repeated many times as the images are generated for display and user either changes magnification or pans/navigates to a different location in the images.

Adaptive image alignment using locally optimal transformations

A system for searching for similar medical images includes a reference library in the form of a multitude of medical images, at least some of which are associated with metadata including clinical information relating to the specimen or patient associated with the medical images. A computer system is configured as a search tool for receiving an input image query from a user. The computer system is trained to find one or more similar medical images in the reference library system which are similar to the input image. The reference library is represented as an embedding of each of the medical images projected in a feature space having a plurality of axes, wherein the embedding is characterized by two aspects of a similarity ranking: (1) visual similarity, and (2) semantic similarity such that neighboring images in the feature space are visually similar and semantic information is represented by the axes of the feature space. The computer system supports additional queries from a user to thereby further refine a search for medical images similar to the input image within a search space consisting of the one or more similar medical images.

Similar medical image search

Growing community of inventors

Belmont, CA, United States of America

Martin Christian Stumpe