Rollback recovery with data lineage capture for data pipelines

Abstract

Computer-readable media, methods, and systems are disclosed for performing rollback recovery with data lineage capture for data pipelines. A middle operator receives ingested input events from a source operator reading data from an external input data source. The middle operator then logs information regarding middle input events to a middle operator input log, designating the logged middle input event information as incomplete. The middle operator then processes data associated with the middle input events and updates the middle input log entries setting them to a completed logging status designation for middle input events that were consumed to produce the one or more middle output events. The middle operator then transmits the middle output events to subsequent operators. Garbage collection is performed to remove completed entries from the middle operator output log. Finally, based on receiving a recovering message from a subsequent operator, corresponding middle output events are re-sent.