Ensuring data quality through deployment automation in data streaming applications

Abstract

Data streaming applications may need to provide high reliability, particularly depending on the nature of the data being streamed. A framework is described that allows a data streaming application to ensure high reliability both during update operations and during ordinary operations. A unique event ID count can be recorded that reflects messages being sent from a source to the streaming application. After an update and service restart, the count can again be collected to see if data is flowing through the streaming application as expected. Unique database record counts can be reviewed (e.g. after a restart or during ordinary operations) to ensure that no records are being unexpectedly dropped. Data content sampling can also be performed to see that any data transformations are functioning properly. Corrective actions (after a restart or during ordinary operations) can also be taken, including replay of database messages that are dropped, or sending an alert.