International Business Machines Corporation

Merative US L.p.

Aviad Zlotnick

Michael Edward Factor

Shachar Fienblit

Eitan D Farchi

Olympia Gluck

Orna Raz

Gail Andrea Spear

Thomas Charles Jarvis

Rachel Tzoref-Brill

Martin Tross

William Frank Micka

Itzhack Goldberg

Shmuel Ur

Itai Segall

Marcel Zalmanovici

Orna Raz-Pelleg

Murray A Reicher

Joseph S Hyde, Ii

Orit Nissan-Messing

Robert Francis Bartfai

Howard M Hess

Rachel Tzoref

Yarden Nir-Buchbinder

Ronen Shevach

Rhonda L Childress

Neil Sondhi

Kenneth Fairclough Day, Iii

John Jay Wolfgang

Zorik Machulsky

Henry Esmond Butterworth

Elad Yom-Tov

Sam Clark Werner

Amiram Hayardeny

Akram Bitar

Flora Gilboa-Solomon

Ehud Trainin

Kenneth Wayne Boyd

Dan S Pelleg

Eyal Gordon

Philip Matthew Doatmas

Yoel Shoshan

Kenneth Nagin

Vadim Ratner

Onn Menahem Shehory

Moshe Klausner

Evgeny Schmeilin

Sergey Novikov

Edward Hsiu-Wei Lin

Amir Kredi

Yoram Simha Adler

Sergey Krutyolkin

Rachel Brill

Eugene Wallach

Alon Hazan

Ziva Sommer

Oren Kagan

Nir Golan

Irit Dolev

Vitaly Zautner

Gregory E McBride

Carol Santich Mellgren

Robert Akira Kubo

Robert Bruce Nicholson

William James Scales

Warren Keith Stanley

Shmuel Ben-Yehuda

Robert Frederic Kern

Abel Alkon Gordon

Lawrence Carter Blount

Kenneth F Day, Iii

Ehud Dov Karnin

Paula Kim Ta-Shma

Gal Ashour

Guy Laden

Andre Heilper

Gary Valentin

John Earle Lindley

Eli Malul

Dana Levanony

Sheli Rahav

Ella Barkan

Alan Hartman

Dalit Tzafrir

Gilad Sharaby

Shlomit Sarah Pinter

Rami Ben-Ari

Efrat Hexter

Hana Chockler

Shira Ben-Dor

John Thomas Flynn, Jr

Paul Wojciak

Guy Amit

Avi Teperman

Daniel Citron

Ayelet Akselrod-Ballin

Muli Ben-Yehuda

Svetlana Shukevich

Moran Shochat

Andrei Kirshin

Sivan Ravid

Outlier detection using a Deep Neural Network (DNN) includes running a trained DNN model on an received input item. A first feature vector is extracted from the input item and quantized to discrete values. A first number of special t-way feature combinations are computed in the input item and compared against a computed threshold. Based on the comparison, the input item is flagged as an outlier and an alert is generated notifying of the flagged input item.

Outlier detection in a deep neural network using t-way feature combinations

There is provided a computer-implemented method of identifying a finding in an input sub-dataset of an input dataset using an ensemble of machine learning (ML) models, comprising: obtaining outcomes of an ensemble of ML models generated in response to feeding each input sub-dataset of the input dataset into each ML model of the ensemble trained to generate an outcome indicative of likelihood of a finding depicted in a respective input sub-dataset, creating a two dimensional (2D) outcome dataset storing outcomes of the ensemble, wherein a first dimension denotes the respective ML model and a second dimension denotes the respective input sub-dataset, computing a filtered 2D dataset by applying a filter function to the 2D outcome dataset, and analyzing the filtered 2D dataset to identify specific input sub-dataset(s) likely depicting the finding.

Identifying a finding in a dataset using a machine learning model ensemble

A method and system for reducing the number of colors per pixel present in an image to increase the ability to detect objects or anomalies in the image. A final number of colors per pixel to reduce the image to is determined, wherein the final number of colors is a number of colors less than the number of colors per pixel in the original image. A corresponding threshold value for each of the final number of colors is identified, such that the corresponding threshold values optimize an arithmetic combination of separation score functions applied to the plurality of pixels and the threshold values. The image is reduced to the final number of colors per pixel, by creating an output image where a value of each output pixel is equal to the number of threshold values that are less than the value of the corresponding input pixel.

Object oriented image normalization

A method, system and computer program product, the method comprising: creating a model representing underperforming cases; from a case collection having a total performance, and which comprises for each of a multiplicity of records: a value for each feature from a collection of features, a ground truth label and a prediction of a machine learning (ML) engine, obtaining one or more features; dividing the records into groups, based on values of the features in each record; for one group of the groups, calculating a performance parameter of the ML engine over the portion of the records associated with the group; subject to the performance parameter of the group being below the total performance in at least a predetermined threshold: determining a characteristic for the group; adding the characteristic of the group to the model; and providing the model to a user, thus indicating under-performing parts of the test collection.

Method and apparatus for enhancing effectivity of machine learning solutions

Embodiments may include novel techniques to improve detection of objects in images, for example, in Digital Breast Tomography and that are applicable to ensembles of detectors. For example, a method may comprise generating a plurality of candidate bounding boxes for each of a plurality of image slices of imaged tissue, each generated candidate bounding box having a probability score, collecting at least some of the generated candidate bounding boxes for each slice, generating a heat map of the filtered candidate bounding boxes and filtering the candidate bounding boxes in the heat map based on a first threshold of values in the heat map, performing Non-Maximum Suppression on the heat map filtered candidate bounding boxes, and outputting at least one bounding box based on the probability score.

Pre-NMS filtering using a heat map

There is provided a method for monitoring interaction with 3D medical images, comprising: dividing the 3D image into a sequence of a 2D images, arranging the sequence into slabs each including at least one 2D image, computing, for each respective slab, a minimal amount of viewing time a user is predicted to spend viewing the respective slab, monitoring, while the 3D medical image is presented on a display, an amount of time a user actually spent viewing portions of the 3D medical image corresponding to each of the of slabs, in response to the amount of time spent viewing a certain portion of the 3D medical image being less than the computed minimal amount of viewing time of a certain slab corresponding to the certain portion, generating instructions for implementation by a user interface indicative of the amount of time spent being less than the computed minimal amount of time.

Machine learning methods for monitoring a user's interaction with 3D medical images

There is provided a system and a method of generating an annotated structured dataset, comprising: receiving a medical classification term, searching over the unstructured patient data for extracting unclassified unstructured text fragments, presenting a subset of the unclassified unstructured text fragments, receiving an indication of a selection of none or at least one of the text fragments, and one of: (i) classifying non-selected unclassified unstructured text fragments according to the medical classification term, and classifying selected text fragments as not satisfying the medical classification term, and (ii) classifying selected unclassified unstructured text fragments according to the medical classification term, and classifying non-selected unclassified unstructured text fragments as not satisfying the medical classification term, and iterating the searching, and/or the presenting, until no text fragments are obtained by the search, wherein the annotated structured dataset is created by the classification of unclassified unstructured text fragments into the medical classification term.

Systems and methods for targeted annotation of data

There is provided a computer implemented method for generating summary images from 3D medical images, comprising: receiving a 3D medical image, dividing the 3D medical images into a sequence of a plurality 2D images, computing a similarity dataset indicative of an amount of similarity between each pair of the plurality of 2D images, segmenting the similarity dataset into a plurality of slabs by minimizing the amount of similarity between consecutive slabs and maximizing the amount of similarity within each slab, aggregating, for each respective slab, the plurality of 2D images into a respective summary image, and presenting on a display, the respective summary image.

Systems and methods for generating summary medical images

Systems and techniques are disclosed for improvement of machine learning systems based on enhanced training data. An example method includes providing a visual concurrent display of a set of images of features, the features requiring classification by a reviewing user. The user interface is provided to enable the reviewing user to assign classifications to the images, the user interface being configured to create, read, update, and/or delete classifications. The user interface is responsive to the user, with the user response indicating at least two images with a single classification. The user interface is updated to represent the single classification.

Systems and user interfaces for enhancement of data utilized in machine-learning based medical image review

A method, apparatus and a computer program product for automated data slicing based on an Artificial Neural Network (ANN). The method comprising: obtaining an ANN, wherein the ANN is configured to provide a prediction for a data instance, wherein the ANN comprises a set of nodes having interconnections therebetween; determining an attribute vector based on a subset of the nodes of the ANN; determining, based on the attribute vector, a plurality of data slices; obtaining a testing dataset comprising testing data instances; computing, for each data slice, a performance measurement of the ANN over the data slice, wherein said computing is based on an application of the ANN on each testing data instance that is mapped to the data slice; and performing an action based on at least a portion of the performance measurements of the data slices.

Automated data slicing based on an artificial neural network

An example system includes a processor to train a neural network model using annotated training data to generate features. The processor is to select a feature vector of the neural network model. The processor is to execute an inference stage on the annotated training data via the neural network model to generate a first set of values corresponding to the annotated training data for features in the selected feature vector. The processor is to execute the inference stage on unannotated data to generate a second set of values corresponding to the unannotated data for the features in the selected feature vector. The processor is to select an item in unannotated data that matches an uncovered combination of feature values in the annotated training data. The processor is to send the selected item for annotation and receive a corresponding additional annotated item to be added to the annotated training data.

Automated selection of unannotated data for annotation based on features generated during training

Embodiments of the present systems and methods may provide improved machine learning performance even though data drift has occurred. For example, a method may comprise providing a machine learning model in a computer system, operating the machine learning model using a first dataset to obtain results of the first dataset, operating the machine learning model using a second dataset to obtain results of the second dataset, performing statistical testing on a confidence distribution of results of the first dataset and of results of the second dataset to determine a difference in a result confidence distribution between the first dataset and of the second dataset, and determining whether data included in the second dataset has data drift relative to the first dataset based on the difference in a result confidence distribution between the first dataset and of the second dataset.

Classifier confidence as a means for identifying data drift

A method and a computer program product for identification and improvement of machine learning (ML) under-performance The method comprises slicing data of ML model based on a functional model representing requirements of a system utilizing the ML model. The functional model comprises a set of attributes and respective domain of values. Each data slice is associated with a different valuation of one or more attributes of the functional model. Each data instance of the ML model is mapped to one or more data slices, based on valuation of the attributes. A performance measurement of the ML model over is computed for each data slice, based on an application of the ML model on each data instance that is mapped to the data slice. A Determination whether ML model adheres to a target performance requirement may be performed based on the performance measurements of the data slices.

Data slicing for machine learning performance testing and improvement

Systems and techniques are disclosed for improvement of machine learning systems based on enhanced training data. An example method includes accessing a database storing associations between objects included in medical images and classifications of the objects. A risk assessment model adapted to determine a risk condition for an object is accessed, the assessment based on features of the object. Risk conditions associated with respective objects are determined based on the risk assessment model. A group of objects associated with a first risk condition is identified. An interactive user interface is generated for display, the user interface concurrently displaying images of the group of objects. The interactive user interface enables a user to select subsets of images to be concurrently assigned a user-selected classification. User selected classifications are provided to a machine learning system adapted to update the risk assessment model based on the classifications to increase accuracy of the model.

Growing community of inventors

Mitzpeh Netofah, Israel

Aviad Zlotnick