Maintaining streaming parity in large-scale pipelines

Abstract

In a pipeline, data events generated by a producer application are temporally grouped by using a group identification tag. For each data event, data points are generated and uploaded to a storage and cache at each point of production and consumption. The storage allows a matching of data events between the production point and the consumption point, thereby ensuring that streaming parity is maintained. In cases of mismatch, the cache allows for detecting missing data events, i.e., identifying data events that were generated by an upstream producer application, but not consumed by a downstream consumer. While being agnostic to the transformations applied by the various applications in the pipeline, the embodiments disclosed herein keep track of the output data events and input data events and precisely identify the missing data events.