Concept shift detection and correction using probabilistic models and learned feature representations

Abstract

Techniques for concept shift detection and correction using probabilistic models and learned feature representations are described. A gaussian process model is trained using representations generated by a primary machine learning (ML) model for existing training data elements in a training memory. For a new batch of data elements, representations again generated by the primary ML model can be used as input for the gaussian process model to generate predictive distributions. When the true targets for the new data elements are not sufficiently likely according to the corresponding predictive distributions, concept shift is likely and the training memory can be purged of the existing data elements before further retraining of the primary ML model.