System and method for managing a data pipeline using a digital twin

Abstract

Methods and systems for managing operation of a data pipeline are disclosed. To manage the data pipeline, a system may include one or more data sources, a data repository, and one or more downstream consumers. Updates to the data pipeline may cause the data pipeline to become misaligned. To avoid misalignment and, therefore, failure of the data pipeline, any updates intended for implementation in the data pipeline may first be implemented in a digital twin of the data pipeline. The digital twin and the data pipeline may operate concurrently using substantially identical data. The performance of the digital twin following implementation of the potential update may be compared to the performance of the data pipeline without implementation of the potential update to obtain a performance delta. If the performance delta meets a threshold, the potential update may cause failure of the data pipeline and may be remediated prior to implementation.