Handling expiration of resources allocated by a resource manager running a data integration job

Abstract

A method includes: receiving, by a computer device, resource request for a data integration job, wherein the resource request is received from a job executor module and defines processes of the data integration job; allocating, by the computer device, containers for the processes of the data integration job; launching, by the computer device, a respective wrapper script on each respective one of the containers after allocating the respective one of the containers; and transmitting, by the computer device and in response to the allocating, node details to the job executor module. In embodiments, the wrapper script running on the container is configured to repeatedly check a predefined location for process commands from a job executor. After the resource manager allocates all the containers for a data integration job according to a resource request, the job executor writes the process commands to the predefined location. Each wrapper script continues to check the predefined location for the process command that it is assigned to run, and runs the process command as soon as it is available at the predefined location. The process commands may be indexed with index values matching those assigned to respective ones of the wrapper scripts.