Generating a data pipeline in an interactive pipeline session utilizing a dedicated computing cluster to access secure data

Abstract

The disclosed systems provide an interactive pipeline session that provides a secure environment to generate and test a data pipeline prior to deployment, utilizing secure data that the data pipeline will utilize upon deployment. The disclosed systems generate a dedicated computing cluster based on receiving a request to initiate an interactive pipeline session within the integrated development environment and authenticate the dedicated computing cluster, which provides the dedicated computing cluster with access to secured data at a secure data source. The disclosed systems can then execute the interactive pipeline session by utilizing the dedicated computing cluster to generate a data pipeline for secure data. Additional features and advantages of one or more embodiments of the present disclosure are outlined in the description which follows, and in part will be obvious from the description, or may be learned by the practice of such example embodiments.