Configurable pipelines for training and deploying machine learning processes in distributed computing environments

Abstract

The disclosed embodiments include computer-implemented processes and systems that establish configurable pipelines for training and deploying machine-learning processes in distributed computing environments. For example, an apparatus may obtain configuration data associated with a plurality of application engines, and pipelining data characterizing a sequential execution of at least a subset of the application engines. Based on the pipelining data, the apparatus may execute sequentially a subset of the application engines in accordance with the configuration data, which may cause the apparatus to perform operations that at least one of (i) train a machine-learning process or (ii) apply the trained machine-learning process to an input dataset. The apparatus may obtain artifact data generated by the executed subset of the application engines, store the artifact data within a data repository, and transmit a portion of the artifact data to a computing system that generated at least one element of the configuration data.