Malicious traffic detection with anomaly detection modeling

Abstract

An anomaly detection model is trained to detect malicious traffic sessions with a low rate of false positives. A sample feature extractor extracts tokens corresponding to human-readable substrings of incoming unstructured payloads in a traffic session. The tokens are correlated with a list of malicious traffic features and frequent malicious traffic features across the traffic session are aggregated into a feature vector of malicious traffic feature frequencies. An anomaly detection model trained on feature vectors for unstructured malicious traffic samples predicts the traffic session as malicious or unclassified. The anomaly detection model is trained and updated based on its' ongoing false positive rate and malicious traffic features in the list of malicious traffic features that result in a high false positive rate are removed.