Robust whitelisting of legitimate files using similarity score and suspiciousness score

Abstract

A locality-sensitive hash value is calculated for a suspect file in an endpoint computer. A similarity score is calculated for the suspect hash value by comparing it to similarly-calculated hash values in a cluster of known benign files. A suspiciousness score is calculated for the suspect hash value based upon similar matches in a cluster of benign files and a cluster of known malicious files. These similarity score and the suspiciousness score or combined in order to determine if the suspect file is malicious or not. Feature extraction and a set of features for the suspect file may be used instead of the hash value; the classes would contain sets of features rather than hash values. The clusters may reside in a cloud service database. The suspiciousness score is a modified Tarantula technique. Matching of locality-sensitive hashes may be performed by traversing tree structures of hash values.