Method and system for compression and optimization of in-line and in-transit information security data

Abstract

This document discloses a method and system for just-in-time compression and optimization of raw unstructured in-line and in-transit data by identifying low entropy data blocks or duplicated information security information in raw computer security alerts within a series of time windows. In particular, the method and system automatically manages; processes; and optimizes in-line and in-transit data blocks or raw information security alerts received from a plurality of information surveillance sources and/or peripheral monitoring devices simultaneously. The data blocks or raw information security alerts that are found to be unique in the various time windows are transposed into meta-definition tables to be further processed while redundant data blocks or raw alerts contained within each particular time window are identified, marked and processed accordingly.