Continuous flow compute point based data processing

Abstract

A data processing system and method that provides two processes, checkpointing and compute point propagation, and permits a continuous flow of data processing by allowing each process to (1) return to normal operation after checkpointing or (2) respond to receipt of a compute point indicator, independently of the time required by other processes for similar responsive actions. Checkpointing makes use of a command message from a checkpoint processor that sequentially propagates through a process stage from data sources through processes to data sinks, triggering each process to checkpoint its state and then pass on a checkpointing message to connected “downstream” processes. A compute point indicator marks blocks of records that should be processed as a group within each process. A compute point indicator is triggered and sequentially propagates through a process stage from data sources through processes to data sinks without external control. Compute point indicators also effectively self-synchronize multiple data flows without external control. Use of compute point indicators rather than checkpoints avoids the time delay that saving state imposes, while permitting a continuous flow of data processing, including outputting results.