Optimizing signature computation and sampling for fast adaptive similarity detection based on algorithm-specific performance

Abstract

A set of similarity detection algorithms and techniques for determining which signature calculation, sampling, and generation algorithms may be most beneficially applied to application related data are described herein. These algorithms work well with SSD caching software to product high speed, high accuracy, and low false-positive detections. Because the different algorithms may show different performance depending on data sets and different applications, to achieve optimal performance, a calibration process may be applied to each application and associated data set to select the best combination of signature computation and sampling technique. The new algorithms are also very fast with execution times an order of magnitude smaller than existing techniques. While some of the algorithms are presented using examples for the purpose of easy readability, these algorithms are very general and can be easily applied to broad range of cases.