Google Inc.

Shravya Shetty

Scott McKinney

Jeffrey De Fauw

Marcin Sieniek

Alex Starns

Varun Godbole

Atilla Kiraly

Natasha Antropova

Christopher Kelly

Jonathan Godwin

Hormuz Mostofi

Bokyung Choi

Diego Ardila

Sujeeth Bharadwaj

Chace Lee

Ryan Gomes

Daniel Tse

Scott Mckinney

A system is described for generating diagnostic information from a video sequence of ultrasound images acquired in 'blind sweeps', i.e., without operator seeing ultrasound images as they are acquired. We disclose two different types of machine learning systems for predicting diagnostic information: a “Temporal Accumulation” system and a “3-D Modeling Component” system. These machine learning systems could be implemented in several possible ways: using just one or the other of them in any given implementation, or using both of them in combination. We also disclose a computing system which implements (a) an image selection system including at least one machine learning model trained to identify clinically suitable images from the sequence of ultrasound images and (b) an image diagnosis/measurement system including of one or more machine learning models, configured to obtain the clinically suitable images identified by the image selection system and further process such images to predict health states.

Automated maternal and prenatal health diagnostics from ultrasound blind sweep video sequences

A method is disclosed of processing a set of images. Each image in the set has an associated counterpart image. One or more regions of interest (ROIs) are identified in one or more of the images in the set of images. For ROI identified, a reference region is identified in the associated counterpart image. ROIs and associated reference regions are cropped out, thereby forming cropped pairs of images 1 . . . n, that are fed to a deep learning model trained to make a prediction of probability of a state of the ROI, e.g., disease state, which generates a prediction P-, (i=1 . . . n) for each cropped pair. The model generates an overall prediction P from each of the predictions P. A visualization of the set of medical images and the associated counterpart images including the cropped pair of images is generated.

System and method for interpretation of multiple medical images using deep learning

A method and system to generate a probabilistic prediction of the presence/absence of cancer in longitudinal and current image datasets, and/or multimodal image datasets, and the location of the cancer, is described. The method and system uses an ensemble of deep learning models. The ensemble includes a global model in the form of a 3D convolutional neural network (CNN) extracting features in the datasets indicative of the presence of cancer on a global basis. The ensemble also includes a two-stage prediction model which includes a first stage or detection model which identifies cancer detection candidates (different cropped volumes of 3D data in the a dataset containing candidates which may be cancer) and a second stage or probability model which incorporates the longitudinal datasets (or multimodal images in a multimodal dataset) and the extracted features from the global model and assigns a cancer probability p to each of the cancer detection candidates. An overall prediction of probability of cancer is obtained from the probabilities assigned by the second stage model, e.g., using a Noisy-OR approach.

Method and system for improving cancer detection using deep learning

A method is provided for processing medical text and associated medical images. A natural language processor configured as a deep conventional neural network is trained on a first corpus of curated free-text, medical reports each of which having one or more structured labels assigned by an medical expert. The network is trained to learn to read additional free-text medical reports and produce predicted structured labels. The natural language processor is applied to a second corpus of free-text medical reports that are associated with medical images. The natural language processor generates structured labels for the associated medical images. A computer vision model is trained using the medical images and the structured labels generated. The computer vision model can thereafter assign a structured label to a further input medical image. In one example, the medical images are chest X-rays.

Automated extraction of structured labels from medical text using deep convolutional networks and use thereof to train a computer vision model

Growing community of inventors

Mountain View, CA, United States of America

Shravya Shetty