Systems, methods and computer program products for scalable, low-latency processing of streaming data

Abstract

A system for processing network flow monitoring data includes a data collection input coupled to at least one incoming data stream; a load balancing operator is operable to distribute the network monitoring messages amongst a plurality of downstream processing units; a plurality of collector processing sub-units downstream from the load balancing operator are operable to generate network flow records from the received network monitoring messages in a first format; a plurality of parser sub-units coupled to the plurality of collector processing sub-units are operable to generate parsed network flow records by converting the received network flow records from the first format to a second format; and a combiner operator coupled to the plurality of parser sub-units is operable to combine the parsed network flow records from the plurality of parser sub-units into a synchronous output data stream.