Early and adaptive stream sampling system and method for machine learning-based optimizations in storage systems

Abstract

Techniques for performing early and adaptive IO stream sampling for ML-based optimizations in a storage system. The techniques include obtaining a sub-slice of sampled data by performing early sampling of a slice of successive operations directed to a storage object. The techniques include generating features based on the sub-slice, processing the features using an ML model, and generating a probability score based on the ML model's output. The techniques include determining that the probability score falls within an overlap range of continuous variable distributions for benign and threat classes of data. The techniques include, in response to the probability score exceeding a specified threshold, comparing a class signature of the sub-slice with a target class signature of the threat class of data to determine a similarity between the class signatures, and, in response to the similarity exceeding a predetermined similarity level, assigning a 'threat' class label to the probability score.