Efficient event correlation in a streaming environment

Abstract

In a streaming environment, efficient correlation between base events and relevant follow-on events is accomplished by temporarily storing events in an in-memory state for a limited time window. The in-memory state buffers the incoming stream of events to permit efficient attempted correlation. Successful correlation (e.g., by ID matching) between a follow-on event and a relevant base event, gives rise to specialized aggregation tables for matched (base, follow-on) event pairs. All events are ultimately removed from the in-memory state upon expiry of the time window. This results in correlated matched event pairs being stored in the aggregation tables for efficient querying. Events remaining uncorrelated upon expiration of the time window, are stored only in an (voluminous) log table for less-efficient (but rare) late querying and aggregation. Efficient correlation is achieved by assuming that successful event pair matching is most likely to occur within the in-memory time window, or not at all.