International Business Machines Corporation

Lukasz Grzegorz Cmielowski

Rafal Bigaj

Wojciech Sobala

Maksymilian Erazmus

Andrzej Jan Wrobel

Tymoteusz Gedliczka

Pawel Slowikowski

Marek Oszajec

Bartlomiej Tomasz Malecki

Umit M Cakmak

Daniel Jakub Ryszka

Marek Franczyk

Marcin Filip

Mateusz S Matejczyk

Kamil Smoron

Gregory Bramble

Blazej Rafal Rutkowski

Maria Hanna Oleszkiewicz

Robert Karcz

Kamila Baron-Palucka

Amadeusz Masny

Szymon Kucharczyk

Dorota Lączak

Disclosed herein is a method of training an artificial intelligence model that comprises an iterative training loop. Said iterative training loop comprises: receiving a current set of training data; dividing said current set of training data into a predetermined number of training data subsets; sequentially training said artificial intelligence model with each of said predetermined number of training data subsets using a training portion and calculating a performance metric using a validation portion; and comparing performance metrics from a previous iteration of said iterative training loop to said calculated performance metric to determine if an improving performance metric condition is met. Said method further comprises halting said iterative training loop unless said improving performance metric condition is not met at least once within a predetermined number of previous iterations.

Early stopping of artificial intelligence model training using control limits

The present disclosure relates to a computer-implemented method for generating a machine learning module (ML-module). The method comprises providing training data comprising a first set of data corresponding to a first feature and a second set of data corresponding to a second feature and generating a further set of data, wherein the further set of data corresponds to a further feature. A first correlation metric as a measure of a correlation between a selected feature of the first and the second feature and the further feature is calculated. Furthermore, a bias metric is determined indicating a strength of a bias of the trained ML-module towards a first subset of values of the further feature or a second subset of values of the further feature if the first correlation metric is greater than a first threshold. If the bias metric satisfies the bias constraint, then the ML-module is released for usage.

Controlling a bias of a machine learning module background

A computer implemented method for detecting regression in a relationship between a performance indicator and AI metrics includes calculating a baseline threshold of regression degradation according to a historical correlation coefficient corresponding to a performance indicator and a set of AI metrics, calculating a current correlation coefficient according to one or more current data records, identifying a correction constant according to the current correlation coefficient and a desired correlation coefficient, generating a function to predict correction constants corresponding to performance indicator data and the set AI metrics, determining a delta correction constant for each AI metric of the set of AI metrics, applying the determined delta correction constant to the set of AI metrics, and identifying a subset of AI metric outliers according to the calculated baseline threshold and the determined delta correction constant.

Regression detection and correction in relationships between performance indicators and ai metrics

A computer-implemented method and computer program product for predicting an impact of an adjustment to a machine learning model to key performance indicators, and a forecasting engine. The computer-implemented method may comprise receiving a proposed adjustment to a machine learning model, calculating, using a regression machine learning model to ingest the proposed adjustment, a set of value components for a key performance indicator (KPI), calculating a plurality of results for the KPI using the set of value components, automatically determining whether the plurality of results exceeds a performance threshold, and recommending the proposed adjustment based on the determination.

Predicting performance of machine learning models

Embodiments of present disclosure relate to a method for predicting a performance of a machine learning module (ML-Module). The method may comprise detecting a change in the performance of the ML-Module over a period of time on the basis of labeled input datasets for the ML-Module and detecting a change in a predicted performance of the ML-Module over the period of time computed using the drift module. A value of a first key figure is determined, the value of the first key figure indicating a correlation between the change in the performance of the ML-Module and the change in the predicted performance of the ML-Module. A signal is provided, the signal indicating the value of the first key figure.

Continuous learning process using concept drift monitoring

An approach for selecting a transformed prediction model for estimating the uncertainty of prediction associated with machine learning is disclosed. The approach determines a function corresponding to a prediction interval based on one or more datasets and calculates one or more sets of prediction interval parameters associated with the function based on training a set of regression models with the one or more datasets. The approach creates one or more transformed predictions based on the one or more sets of parameters and based on a logical distance and selects a transformed prediction model based on a mean absolute correlation from the one or more transformed predictions. Furthermore, the approach outputs the selected transformed prediction model.

Standard error of prediction of performance in artificial intelligence model

A method, computer system, and a computer program product for providing reduced training data for training a machine learning model (ML-model) is disclosed. The present invention may include generating the reduced training data dependent on a first two batches of data records of original training data. The present invention may include generating an updated version of the reduced training data dependent on the reduced training data and a further batch of the original training data, wherein a size of the updated version of the reduced training data is equal or less than a size of the limited storage capacity and less than a combined size of the reduced training data and the further batch together. The reading of a further batch of the data records of the original training data and the generating of an updated version of the reduced training data may be repeated.

Providing reduced training data for training a machine learning model

Identifying a change of an indicator value for a system of interdependent entities includes determining the indicator value, logging input data for the system of interdependent entities, logging scoring payload data and related results of a machine-learning model used as part of the system of interdependent entities, wherein the scoring payload data are derived from the input data, clustering the input data into a number of clusters resulting in defined clusters, determining metric values of the machine-learning model by mapping each of the defined clusters onto the scoring payload data resulting in a vector of metric values, re-determining the indicator value for each defined cluster, resulting in a vector of re-calculated indicator values, and determining correlation matrix values for a matrix between the vector of re-determined indicator values and the vector of the metric values for each of the defined clusters.

Finding root cause for low key performance indicators

A computer-implemented method, system and computer program product for analyzing test result failures using artificial intelligence models. A first machine learning model is trained to differentiate between a bug failure and a test failure within the test failures based on the failure attributes and historical failures. The failure type for each failed test in test failure groups is then determined using the first machine learning model. The failed tests in the test failure groups are then clustered into a set of clusters according to the failure attributes and the determined failure type for each failed test. A root cause failure for each cluster is identified based on the set of clusters and the failure attributes. The root cause of an unclassified failure is predicted using a second machine learning model trained to predict a root cause of the unclassified failure based on identifying the root cause failure for each cluster.

Analyzing test result failures using artificial intelligence models

A dataset is received that is for processing by a machine learning model. A scoring payload for the dataset and that regards the machine learning model is also received. A set of features of the machine learning model is determined by analyzing the scoring payload. The scoring payload is structured in accordance with the set of features such that the structured scoring payload is ready for analysis for a monitor of the machine learning model.

Machine learning model monitoring

An overall performance metric of a computer system may be determined for each bin of the set of analysis bins. In case one or more bins of the set of analysis bins do not have at least a predefined minimum number of records, a new set of analysis bins may be redefined by joining analysis bins of the set of analysis bins. For each bin of the redefined set of bins a machine learning (ML) performance metric of the ML model may be computed. The ML performance metric may be estimated for the set of analysis bins using the ML performance metrics of the redefined bins. The computer system may be configured based on a correlation over the set of analysis bins between the computed overall performance metric and the ML performance metric.

Machine learning based data monitoring

A first machine learning model processes a set of inputs to generate a first set of results. Based on that first set of results, a quality control range is calculated. A second machine learning model calculates a mean accuracy of a second set of results, based on the set of inputs. A determination of whether the mean accuracy of the second set of results is within the quality control range is made, and a user is notified of that determination.

Evaluating surrogate machine learning models

A method, system, and computer program product for correlation detection between artificial intelligence (AI) transactions. The method stores a set of transaction records associated with an AI decision engine. Each transaction record has a set of record characteristics. The method assigns the set of transaction records to a set of batches on the set of record characteristics. A set of batch characteristics are determined for a batch of the set of batches. The method determines one or more correlations among the set of batch characteristics. The one or more correlations are compared with one or more threshold batches. The method determines, from the one or more correlations and the comparing, an impact of one or more recommendations of the AI decision engine. The one or more recommendations are defined by the set of transaction records.

Multi-dimensional record correlations

Aspects of the present invention provide an approach for exposing payloads from non-integrated machine learning systems. A generic binding identifier is established to represent a machine learning (ML) system among a set of non-integrated learning systems. A generic subscription identifier is established to represent a deployed model in the ML system. Payload data including a user request, a response, the generic binding identifier, and the generic subscription identifier are received from the ML system and stored in a database for later analysis to identify any issues related to the deployed model.

Exposing payload data from non-integrated machine learning systems

A method and a computer program product are used generating an index of a scoring payload dataset. Correlation coefficients for correlations between input data values and output data values of the machine learning model provided by the scoring payload datasets as well as performance data values of the processes provided by process datasets are calculated. Features of which feature values are used as input data values are ranked according to their importance using the correlation coefficients. For the features of a set of highest-ranking features feature value sets with feature values of the respective features are selected from the scoring payload datasets and a database index of the selected feature value sets is generated.

Growing community of inventors

Cracow, Poland

Lukasz Grzegorz Cmielowski