System and method for determining and maintaining data quality in data processing

Abstract

Systems, computer program products, and methods for determining and maintaining data quality in data processing is provided. The method includes determining one or more plugins used in a flow execution. The flow execution performs at least one transformation on one or more data sets. The method also includes determining one or more automatic data quality rules based on at least one of the one or more plugins using in the flow execution. The method further includes determining one or more custom data quality rules based on a custom data quality rule input received from an end-point device. The custom data quality rule includes one or more data statistics to be monitored during the flow execution. The method still further includes causing execution of each of the automatic data quality rules and the one or more custom data quality rules on data in the flow execution.