International Business Machines Corporation

Marcel Zalmanovici

Orna Raz

Aviad Zlotnick

Eitan D Farchi

Gad Haber

Sergey Novikov

Itzhack Goldberg

Ramani Ranjan Routray

Sheng Hua Bao

Andre Heilper

Moshe Klausner

Sima Nadler

Paul Wojciak

Daniel Citron

Pathirage D S U Perera

Guy Bashkansky

Randall L Tackett

A method, system and computer program product, the method comprising: creating a model representing underperforming cases; from a case collection having a total performance, and which comprises for each of a multiplicity of records: a value for each feature from a collection of features, a ground truth label and a prediction of a machine learning (ML) engine, obtaining one or more features; dividing the records into groups, based on values of the features in each record; for one group of the groups, calculating a performance parameter of the ML engine over the portion of the records associated with the group; subject to the performance parameter of the group being below the total performance in at least a predetermined threshold: determining a characteristic for the group; adding the characteristic of the group to the model; and providing the model to a user, thus indicating under-performing parts of the test collection.

Method and apparatus for enhancing effectivity of machine learning solutions

A method, apparatus and a product for determining a performance measurement of predictors. The method comprises obtaining a dataset comprising data instances. Each data instance is associated with a label; obtaining a predictor. The predictor is configured to provide a prediction of a label for a data instance; determining a plurality of data slices that are subsets of the dataset. computing, for each data slice in the plurality of data slices and based on an application of the predictor on each data instance that is mapped to the data slice, a performance measurement that is indicative of a successful label prediction for a data instance comprised by the data slice, whereby obtaining a plurality of performance measurements; based on the plurality of performance measurements, computing a performance measurement of the predictor over the dataset; if the performance measurement of the predictor is below a threshold, performing a mitigating action.

Performance measurement of predictors

A method, apparatus and product for identifying data drifts. The method comprising: obtaining a baseline dataset of instances in a feature space, each of wherein being associated with a label; determining a set of clusters in the feature space, based on the baseline dataset; determining a baseline distribution of instances over the set of clusters based on the baseline dataset; for each cluster, computing a performance metric for a predictor for the each cluster, wherein the predictor is configured to estimate an estimated label for an instance, wherein the performance metric is indicative of a successful estimation of the predictor to a portion of the baseline dataset that is comprised by the cluster; obtaining a second dataset, wherein the second dataset comprising instances in the feature space; determining a second distribution of instances over the set of clusters, wherein said determining the second distribution is based on the second dataset; and based on the second distribution and on the baseline distribution, and based on at least one performance metric of at least one the cluster of the set of clusters, identifying a data drift in the second dataset with respect to the baseline dataset.

Identifying data drifts that have an adverse effect on predictors

A computer system trains a machine learning model. A vector representation is generated for each document in a collection of documents. The documents are clustered based on the vector representations of the documents to produce a plurality of clusters. A training set is produced by selecting one or more documents from each cluster, wherein the selected documents represent a sample of the collection of documents to train the machine learning model. The machine learning model is trained by applying the training set to the machine learning model. Embodiments of the present invention further include a method and program product for training a machine learning model in substantially the same manner described above.

Generating training sets to train machine learning models

Embodiments of the present systems and methods may provide improved machine learning performance even though data drift has occurred. For example, a method may comprise providing a machine learning model in a computer system, operating the machine learning model using a first dataset to obtain results of the first dataset, operating the machine learning model using a second dataset to obtain results of the second dataset, performing statistical testing on a confidence distribution of results of the first dataset and of results of the second dataset to determine a difference in a result confidence distribution between the first dataset and of the second dataset, and determining whether data included in the second dataset has data drift relative to the first dataset based on the difference in a result confidence distribution between the first dataset and of the second dataset.

Classifier confidence as a means for identifying data drift

A method, a computerized apparatus and a computer program product for mitigating governance and regulation implications on machine learning. A governance impact assessment is generated for a partial data set generated by applying a data governance enforcement on a data set of instances comprising valuations of a feature vector. The partial data set comprises partial instances each comprising partial feature vectors. The governance impact assessment comprises information about data excluded from the data set. A machine learning model trained based on the partial data set and configured to provide an estimated prediction for a partial instance is obtained. A set of core features is determined. A bias introduced by the data governance is identified based on a core feature being affected by the data governance. In response to identifying a bias, an anti-bias procedure is applied on the machine learning model, whereby mitigating the bias introduced by the data governance.

Mitigating governance impact on machine learning

A method, apparatus and computer program product for program migration, the method comprising: receiving a target host and an application to be migrated to a target host; estimating a target load of the application to be migrated; generating a synthetic application which simulates a simulated load, the simulated load being smaller than the target load; loading the synthetic application to the target host; monitoring behavior of the target host, the synthetic application, or a second application executed thereon; subject to the behavior being satisfactory: if the simulated load is smaller than the target load, then repeating said generating, said loading and said monitoring, wherein said loading is repeated with increased load; and otherwise migrating the application to the target.

Safe consolidation and migration

Computer-implemented method, computerized apparatus and computer program product for extracting test model from a textual test suite. The method comprises obtaining a test suite comprising test descriptions. The test descriptions are analyzed to extract attributes and values of a test model modeling a test space. Using the extracted attributes and values, the test model may be created. In some cases, the test model may be partial test model that a user can use as a starting point for manually modeling the textual test suite.

Extracting test model from textual test suite

A method, product and apparatus for creating functional model of test cases. The method comprising obtaining a set of test cases, wherein each test case of the set of test cases comprises free-text; defining one or more tags, wherein each tag of the one or more tags is associated with a query that is configured, when applied, to determine possession of the tag with respect to a test case based on the free-text; applying the queries on the set of test cases to determine possession of the of the one or more tags for each test case; and generating a functional model based on the set of test cases, wherein the functional model comprising for each tag of the one or more tags, a corresponding functional attribute.

Method and system for creating functional model of test cases

A method for flattening conditional statements, the method comprises: obtaining a program code, the program code comprising a conditional control flow program construct, which conditional control flow program construct when read by a target processor, causes the target processor to select a control flow path for execution between at least a first and a second control flow paths, wherein said selection is based on an evaluation of a condition of the conditional control flow program construct; replacing the conditional control flow program construct with a transaction-based control flow program construct, which when read by the target processor is operative to cause the target processor to commence a transaction, the transaction configured to execute the first control flow path; and wherein the transaction-based control flow program construct is operative to cause the target processor to execute the conditional control flow program construct in case the transaction is rolled back.

Flattening conditional statements

Systems and methods for data profiling are provided. In one embodiment, the method comprises monitoring value of at least a target parameter during execution of logic code in a computing environment, wherein the value of the target parameter is incrementally updated in a sequence of data points; and using statistical analysis to determine a target value for the target parameter as of a current data point, in response to determining a change in the value of the target parameter at each data point.

Efficient data profiling to optimize system performance

Device, system, and method of computer program optimization. For example, an apparatus to analyze a plurality of versions of computer program includes: a code analyzer to determine one or more code differences between first and second versions of the computer program, based on at least one optimization log associated with at least one of the first and second versions of the computer program.

Device, system, and method of computer program optimization

A novel and useful apparatus for and method of associating a dedicated coverage bit to each instruction in a software system. Coverage bits are set every time the software application runs, enabling a more comprehensive and on-going code coverage analysis. The code coverage bit mechanism enables code coverage analysis for all installations of a software application, not just software in development mode or at a specific installation. Code coverage bits are implemented in either the instruction set architecture (ISA) of the central processing unit, the executable file of a software application, a companion file to the executable file or a code coverage table residing in memory of the computer system.

Apparatus for and method of life-time test coverage for executable code

A method and system for profiling performance behaviour of executed loops. For each invocation of a loop, a count of a measured event is incremented. A display is provided for a loop () showing the number of measured events for each of the loop's invocations. The code of a loop is instrumented () to obtain the count of loop invocations () and the occurrences of the measured event ().

Growing community of inventors

Kiriat Motzkin, Israel

Marcel Zalmanovici