Systems, methods and computer program products for scalable, low-latency processing of streaming data

Abstract

A system for processing cybersecurity monitoring data includes a load balancing operator coupled to at least one incoming data stream. The load balancing operator is operable to distribute cybersecurity monitoring messages amongst a plurality of collector processing units operable to generate monitoring records from the received cybersecurity monitoring messages in a first format. A plurality of parser sub-units coupled to the plurality of collector processing sub-units are operable to generate parsed monitoring records by converting the received monitoring records from the first format to a second format. A combiner operator coupled to the plurality of parser sub-units is operable to combine the parsed monitoring records from the plurality of parser sub-units into a synchronous output data stream. The system is configured such that data flows sequentially from the load balancing operator to the plurality of collector processing sub-units and to the plurality of parser sub-units to the combiner operator.