Auto-scaling, resilient, and load balancing framework for workload deployment in cloud and high performance computing environments

Abstract

In certain embodiments, a method includes receiving, at a framework daemon, an indication to begin execution of a workload divided into execution units; accessing, by the framework daemon and in response to the indication, an execution context that includes the execution units; associating, by the framework daemon, an identifier of a node with an execution unit of the execution units to claim the execution unit for execution by the node; executing, by the node, the execution unit; accessing, by the framework daemon and after the node completes execution of the execution unit, the execution context to determine whether the execution context includes an unclaimed execution unit; associating, by the framework daemon and when the execution context includes an unclaimed execution unit, the identifier of the node with the unclaimed execution unit to claim the unclaimed execution unit for execution by the node; and executing, by the node, the execution unit.