International Business Machines Corporation

Orna Raz

Eitan D Farchi

Aviad Zlotnick

Marcel Zalmanovici

Ramani Ranjan Routray

Rachel Tzoref-Brill

Sheng Hua Bao

Howard M Hess

Pathirage D Perera

Onn Menahem Shehory

Moshe Klausner

Yoram Simha Adler

Rachel Brill

Pathirage Dinindu Sujan Udayanga Perera

Jinho Hwang

Christopher V Derobertis

Hai Huang

Shripad Nadgowda

Lawrence Carter Blount

Sima Nadler

Hana Chockler

Paul Wojciak

Pathirage D S U Perera

Aharon Brodie

Vivek Krishnamurthy

Samuel Solomon Ackerman

Nakul Chakrapani

Ateret Anaby - Tavor

Matan Vetzler

Randall L Tackett

Yochai Ben-Chaim

Various systems and methods are presented regarding detecting data drift. The data of interest can be batches of utterances received at an interface (e.g., a chatbot). The batches of utterances can be compared with topics present in training data utilized to train a data classifier (e.g., an autoencoder), wherein topics identified in the batches of utterances that are not present in the training data can be considered to be novel topics. The greater the presence of novel topics in a batch of utterances, the greater the divergence of the batch of utterances from the content of the training data. The novel topics can be identified and subsequently applied to the training data such that the data classifier can be re-trained with the novel topics, thereby causing the data classifier to be contemporaneous with the novel topics. In an embodiment, the utterances can be short streams of text, symbols, and suchlike.

Reliable and interpretable drift detection in streams of short texts

A method, system and computer program product, the method comprising: creating a model representing underperforming cases; from a case collection having a total performance, and which comprises for each of a multiplicity of records: a value for each feature from a collection of features, a ground truth label and a prediction of a machine learning (ML) engine, obtaining one or more features; dividing the records into groups, based on values of the features in each record; for one group of the groups, calculating a performance parameter of the ML engine over the portion of the records associated with the group; subject to the performance parameter of the group being below the total performance in at least a predetermined threshold: determining a characteristic for the group; adding the characteristic of the group to the model; and providing the model to a user, thus indicating under-performing parts of the test collection.

Method and apparatus for enhancing effectivity of machine learning solutions

A method, apparatus and a product for determining a performance measurement of predictors. The method comprises obtaining a dataset comprising data instances. Each data instance is associated with a label; obtaining a predictor. The predictor is configured to provide a prediction of a label for a data instance; determining a plurality of data slices that are subsets of the dataset. computing, for each data slice in the plurality of data slices and based on an application of the predictor on each data instance that is mapped to the data slice, a performance measurement that is indicative of a successful label prediction for a data instance comprised by the data slice, whereby obtaining a plurality of performance measurements; based on the plurality of performance measurements, computing a performance measurement of the predictor over the dataset; if the performance measurement of the predictor is below a threshold, performing a mitigating action.

Performance measurement of predictors

A mechanism is provided in a data processing system having a processor and a memory. The memory comprises instructions which are executed by the processor to cause the processor to implement a training system for finding an optimal surface for hierarchical classification task on an ontology. The training system receives a training data set and a hierarchical classification ontology data structure. The training system generates a neural network architecture based on the training data set and the hierarchical classification ontology data structure. The neural network architecture comprises an indicative layer, a parent tier (PT) output and a lower leaf tier (LLT) output. The training system trains the neural network architecture to classify the training data set to leaf nodes at the LLT output and parent nodes at the PT output. The indicative layer in the neural network architecture determines a surface that passes through each path from a root to a leaf node in the hierarchical ontology data structure. The training system trains a classifier model for a cognitive system using the surface and the training data set.

Optimizing hierarchical classification with adaptive node collapses

A method, apparatus and product for identifying data drifts. The method comprising: obtaining a baseline dataset of instances in a feature space, each of wherein being associated with a label; determining a set of clusters in the feature space, based on the baseline dataset; determining a baseline distribution of instances over the set of clusters based on the baseline dataset; for each cluster, computing a performance metric for a predictor for the each cluster, wherein the predictor is configured to estimate an estimated label for an instance, wherein the performance metric is indicative of a successful estimation of the predictor to a portion of the baseline dataset that is comprised by the cluster; obtaining a second dataset, wherein the second dataset comprising instances in the feature space; determining a second distribution of instances over the set of clusters, wherein said determining the second distribution is based on the second dataset; and based on the second distribution and on the baseline distribution, and based on at least one performance metric of at least one the cluster of the set of clusters, identifying a data drift in the second dataset with respect to the baseline dataset.

Identifying data drifts that have an adverse effect on predictors

A method, computer system, and a computer program product for assessing a likelihood of success associated with developing at least one machine learning (ML) solution is provided. The present invention may include generating a set of questions based on a set of raw training data. The present invention may also include computing a feasibility score based on an answer corresponding with each question from the generated set of questions. The present invention may then include, in response to determining that the computed feasibility score satisfies a threshold, computing a level of effort associated with developing the at least one ML solution to address a problem. The present invention may further include presenting, to a user, a plurality of results associated with assessing the likelihood of success of the at least one ML solution.

Estimating feasibility and effort for a machine learning solution

A method, system and computer program product, the method comprising: obtaining computer code of an employed system comprising a plurality of components; obtaining data related to operating the plurality of components; based on the computer code and the data, identifying: a first component from the plurality of components, to be maintained; and a second component from the plurality of components, to be at least partly replaced by a machine learning component; and providing to a user an identification of the first component and the second component.

Method and apparatus for employing machine learning solutions

A method, apparatus and a computer program product for automated data slicing based on an Artificial Neural Network (ANN). The method comprising: obtaining an ANN, wherein the ANN is configured to provide a prediction for a data instance, wherein the ANN comprises a set of nodes having interconnections therebetween; determining an attribute vector based on a subset of the nodes of the ANN; determining, based on the attribute vector, a plurality of data slices; obtaining a testing dataset comprising testing data instances; computing, for each data slice, a performance measurement of the ANN over the data slice, wherein said computing is based on an application of the ANN on each testing data instance that is mapped to the data slice; and performing an action based on at least a portion of the performance measurements of the data slices.

Automated data slicing based on an artificial neural network

A computer system trains a machine learning model. A vector representation is generated for each document in a collection of documents. The documents are clustered based on the vector representations of the documents to produce a plurality of clusters. A training set is produced by selecting one or more documents from each cluster, wherein the selected documents represent a sample of the collection of documents to train the machine learning model. The machine learning model is trained by applying the training set to the machine learning model. Embodiments of the present invention further include a method and program product for training a machine learning model in substantially the same manner described above.

Generating training sets to train machine learning models

Embodiments of the present systems and methods may provide improved machine learning performance even though data drift has occurred. For example, a method may comprise providing a machine learning model in a computer system, operating the machine learning model using a first dataset to obtain results of the first dataset, operating the machine learning model using a second dataset to obtain results of the second dataset, performing statistical testing on a confidence distribution of results of the first dataset and of results of the second dataset to determine a difference in a result confidence distribution between the first dataset and of the second dataset, and determining whether data included in the second dataset has data drift relative to the first dataset based on the difference in a result confidence distribution between the first dataset and of the second dataset.

Classifier confidence as a means for identifying data drift

A method and a computer program product for identification and improvement of machine learning (ML) under-performance The method comprises slicing data of ML model based on a functional model representing requirements of a system utilizing the ML model. The functional model comprises a set of attributes and respective domain of values. Each data slice is associated with a different valuation of one or more attributes of the functional model. Each data instance of the ML model is mapped to one or more data slices, based on valuation of the attributes. A performance measurement of the ML model over is computed for each data slice, based on an application of the ML model on each data instance that is mapped to the data slice. A Determination whether ML model adheres to a target performance requirement may be performed based on the performance measurements of the data slices.

Data slicing for machine learning performance testing and improvement

From metadata corresponding to a narrative text, a first encoding is constructed, the first encoding comprising a standardized text string, the first encoding formed according to an encoding scheme. A specified portion of the standardized text string of the first encoding is marked as an anchor term. A correspondence between the first encoding and a second encoding is tested using the encoding scheme and a Natural Language Processing engine, responsive to finding the anchor term within the narrative text. The second encoding corresponds to a text window. The text window comprises a portion of the narrative text comprising an instance of the anchor term and a word within a predetermined distance from the instance. Responsive to the second encoding being identical to the first encoding, the narrative text is annotated, the annotating creating new data linking the narrative text with the second encoding.

Encoding-assisted annotation of narrative text

A mechanism is provided in a data processing system having a processor and a memory. The memory comprises instructions which are executed by the processor to cause the processor to implement a training system for finding an optimal surface for hierarchical classification task on an ontology. The training system receives a training data set and a hierarchical ontology data structure. A surface finding component executing within the training system selects a surface that passes through each path from a root to a leaf node in the hierarchical ontology data structure. The surface finding component determines a plurality of adjacent surfaces that differ from the selected component by one node. The surface finding component selects an optimal surface, based on the selected surface and the plurality of adjacent surfaces, that maximizes accuracy and coverage. The training system trains a classifier model for a cognitive system using the optimal surface and the training data set.

Finding optimal surface for hierarchical classification task on an ontology

A method, a computerized apparatus and a computer program product for mitigating governance and regulation implications on machine learning. A governance impact assessment is generated for a partial data set generated by applying a data governance enforcement on a data set of instances comprising valuations of a feature vector. The partial data set comprises partial instances each comprising partial feature vectors. The governance impact assessment comprises information about data excluded from the data set. A machine learning model trained based on the partial data set and configured to provide an estimated prediction for a partial instance is obtained. A set of core features is determined. A bias introduced by the data governance is identified based on a core feature being affected by the data governance. In response to identifying a bias, an anti-bias procedure is applied on the machine learning model, whereby mitigating the bias introduced by the data governance.

Growing community of inventors

Haifa, Israel

Orna Raz