Distributed data processing and machine learning workload scaling

Abstract

Systems and techniques may generally be used for streamlining a deployment and scaling of data processing and machine learning workloads on a distributed system. An example method may include receiving, from a user at a user interface, a plugin command, an input including a name key and a query, and a driver code including a code package. The method may include filtering the query based on a scope of the plugin command, obtaining input data by querying an input measure group, and slicing the obtained input data into one or more slices based on the slicing key. The method may include determining a number of containers to be used, and assigning at least one slice for each container. The method may include executing at each container the respective assigned at least one slice, generating an output for each input, and storing each output at a respective output measure group.