Consistent data delivery in an eventually consistent system

Abstract

Certain aspects of the disclosure provide techniques for delivering consistent data in an eventually consistent system. An example method includes storing a first data table comprising a trigger timestamp; storing a second data table comprising a commit timestamp and an adaptor offset corresponding to a latest capture change data (CDC) event recorded in a categorized data stream; determining that the commit timestamp is later than or the same as the trigger timestamp; storing a third data table comprising a file writer offset corresponding to a latest data file stored in a file storage system; determining that the file writer offset is greater than or equal to the adaptor offset; causing the data files stored in the file storage system to be transmitted to and stored in a data lake; and generating and transmitting a notification comprising an indication that the data lake is up-to-date as of the trigger timestamp.