Automatic expected validation definition generation for data correctness in ai/ml pipelines

Abstract

Systems and methods are provided for generating extract-transform-load ('ETL') machine learning (“ML”) pipeline validation rules based on user-input, wherein the ETL ML pipeline validation rules may be applicable to validate an ETL ML pipeline against multiple test datasets. The ETL ML pipeline validation rules may comprise compute-type validation rules for computing expected values of data structures within a dataset output by the ETL ML pipeline. The ETL ML pipeline validation rules may comprise check-type validation rules for checking whether data structures within a dataset output by the ETL ML pipeline have intended characteristics. Where the ETL ML pipeline validation rules are applicable to validate an ETL ML pipeline against a test dataset which was not referenced to describe the ETL ML pipeline validation rules, then the ETL ML pipeline may reuse these ETL ML pipeline validation rules to validate the ETL ML pipeline without further user-input.