Diagnosing anomalies detected by black-box machine learning models

Abstract

A computer-implemented method, a computer program product, and a computer system for diagnosing anomalies detected by a black-box machine learning model. A computer determines a local variance of a test sample in a test dataset, where the local variance represents uncertainty of a prediction by the black-box machine learning model. The computer initializes optimal compensations for the test sample, where the optimal compensations are optimal perturbations to test sample values of respective components of a multivariate input variable. The computer determines local gradients for the test sample. Based on the local variance and the local gradients, the computer updates the optimal compensations until convergences of the optimal compensations are reached. Using the optimal compensations, the computer diagnoses the anomalies detected by the black-box machine learning model.