Sas Institute Inc.

Eric Jian Yang

Henry Gabriel Victor Bequet

Ronald Earl Stogner

Qing Gong

Chaowang “Ricky” Zhang

Kais Arfaoui

Partha Dutta

Chaowang Ricky Zhang

An apparatus includes a processor to receive a request to provide a view of an object associated with a job flow, and in response to determining that the object is associated with a task type requiring access to a particular resource not accessible to a first interpretation routine: store, within a job queue, a job flow generation request message to cause generation of a job flow definition the defines another job flow for generating the requested view; within a task container in which a second interpretation routine that does have access to the particular resource is executed, generate the job flow definition; store, within a task queue, a job flow generation completion message that includes a copy of the job flow definition; use the job flow definition to perform the other job flow to generate the requested view; and transmit the requested view to the requesting device.

Automated job flow generation to provide object views in container-supported many task computing

An apparatus including a processor to: output a first request message onto a group sub-queue shared by multiple task containers to request execution of a first task routine; within a task container, respond to the first request message, by outputting a first task in-progress message onto an individual sub-queue not shared with other task containers to accede to executing the first task routine, followed by a task completion message; and respond to the task completion message by allowing the task completion message to remain on the individual sub-queue to keep the task container from executing another task routine from another request message on the group sub-queue, outputting a second request message onto the individual sub-queue to cause execution of a second task routine within the same task container to perform a second task, and responding to the second task in-progress message by de-queuing the task completion message.

Message queue protocol for sequential execution of related task routines in many task computing

An apparatus including a processor to: within a kill container, in response to a set of error messages indicative of errors in executing multiple instances of a task routine to perform a task of a job flow with multiple data object blocks of a data object, and in response to the quantity of error messages reaching a threshold, output a kill tasks request message that identifies the job flow; within a task container, in response to the kill tasks request message, cease execution of the task routine and output a task cancelation message that identifies the task and the job flow; and within a performance container, in response to he task cancelation message, output a job cancelation message to cause the transmission of an indication of cancelation of the job flow, via a network, and to a requesting device that requested the performance of the job flow.

Automated job flow cancellation for multiple task routine instance errors in many task computing

An apparatus includes a processor to: output a request message to cause a first task to be performed; within a task container, in response to the request message and a data object not being divided, divide the data object into a set of data object blocks based on at least the sizes of the data object and the atomic unit of organization of data therein, as well as the storage resources allocated to task containers, and output a task completion message indicating that the first task has been performed, and including a set of data block identifiers indicating the location of the set of data object blocks within at least one federated area; and in response to the task completion message, output a set of request messages to cause a second task to be performed by executing multiple instances of a task routine within multiple task containers.

Data object preparation for execution of multiple task routine instances in many task computing

An apparatus includes a processor to: receive a request to perform a job flow; within a performance container, based on the data dependencies among a set of tasks of the job flow, derive an order of performance of the set of tasks that includes a subset able to be performed in parallel, and derive a quantity of task containers to enable the parallel performance of the subset; based on the derived quantity of task containers, derive a quantity of virtual machines (VMs) to enable the parallel performance of the subset; provide, to a VM allocation routine, an indication of a need for provision of the quantity of VMs; and store, within a task queue, multiple task routine execution request messages to enable parallel execution of task routines within the quantity of task containers to cause the parallel performance of the subset.

Automated virtual machine resource management in container-supported many task computing

An apparatus includes a processor to: within a first container, and prior to its uninstantiation, execute a first instance of a routine to cause the processor to monitor for and detect a job performance request in a job queue; and within a second container, execute a second instance of the routine to cause the processor to search the job queue for a job performance request, and in response to a combination of the uninstantiation of the first container, the storage of the job performance request in the job queue and there being no indication of completion of the job flow in the job queue, perform a combination of store an indication of the job flow performance commencing in the job queue, derive an order of performance of the set of tasks of the job flow and store a first task execution request in a task queue.

Coordinated performance controller failover in many task computing

An apparatus includes a processor to: within a performance container, execute a performance routine to derive an order of performance of tasks of a job flow based on dependencies, begin performing the tasks, and store, within a job queue, a job performance status indication including task performance statuses; identify a set of sub flows within the job flow based on branches in the job flow; correlate each of the task performance statuses to a corresponding sub flow performance status; reduce the job performance status indication size by, for each sub flow in which all tasks have been completed, replace the corresponding task performance statuses with the corresponding sub flow performance status of completed, and for each sub flow with no task performed, replace the corresponding task performance statuses with the corresponding sub flow performance status of not executed; and transmit the job performance status indication to the requesting device.

Implicit status in many task computing

An apparatus includes a processor to: based on data dependencies specified in a job flow definition, identify first and second tasks of the corresponding job flow to be performed sequentially, wherein the first task outputs a data object used as an input to the second; store, within a task queue, at least one message conveying at least an identifier of the first task, and an indication that the data object is to be exchanged through a shared memory space; within a task container, in response to storage of the at least one message within the task queue, sequentially execute first and second task routines to sequentially perform the first and second tasks, respectively, and instantiate the shared memory space to be accessible to the first and second task routines during their executions; and upon completion of the job flow, transmit an indication of completion to another device via a network.

Exchange of data objects between task routines via shared memory space

An apparatus includes a processor to: use an identifier of a requesting device or operator thereof to identify federated area(s) to which access is authorized; based on data dependencies among a set of tasks of a job flow, derive an order of performance specifying the first task to be performed; store, within a task queue, a task routine execution request message including an identifier associated with the first task, and federated area identifier(s) of the identified federated area(s); within a resolver container, in response to storage of the task routine execution request message, use the identifier associated with the first task and identifier(s) of the federated area(s) to identify one in which a first task routine is stored; within a task container, execute the first task routine to perform the first task; and upon completion of the job flow, transmit an indication of completion to the requesting device.

Per task routine distributed resolver

An apparatus includes at least on processor to: parse a job flow definition for a job flow to identify an implicit expression of a data dependency arising from a data object output by a first task of the job flow and input to a second task thereof, wherein the first task is referred to as a data object required by the second as an input; in response to identifying the implicit expression, derive an order of performance of the tasks of the job flow that includes performing the first task before the second to ensure generation of the data object prior to performance of the second; for each task, retrieve a corresponding task routine; execute the task routines in an order that follows the order of performance of the tasks; and transmit, to the requesting device via the network, an indication of successful performance of the job flow.

Automated message-based job flow resource coordination in container-supported many task computing

An apparatus includes processor(s) to: within a kill container, monitor a task kill queue for execution failure message(s) indicating failure of a task routine to perform a task of a job flow, and in response to a quantity of execution failure messages reaching a threshold, store, within the task kill queue, a kill tasks request message; within a task container, in response to the kill tasks request message, cease execution of the task routine to cease performance of the task, and store, within the task queue, a task cancelation message indicating cancelation of the task routine; within a performance container, in response to the task canceled message, store, within a job queue, a job cancelation message indicating cancelation of the job flow; and in response to the job cancelation message, transmit an indication of cancelation of the job flow to a requesting device.

Automated message-based job flow cancellation in container-supported many task computing

An apparatus includes at least one processor to: within a kill container, execute a kill routine to monitor a task kill queue for storage of an execution status message indicating a level of a parameter of execution of a task routine to perform a task of a job flow, and in response to the level exceeding a threshold, store, within the task kill queue, a kill tasks request message; within a task container, in response to the kill tasks request message, cease execution of the task routine, and store, within a task queue, a task cancelation message; within a performance container, execute instructions of a performance routine to, in response to the task cancelation message, store, within a job queue, a job cancelation message; and in response to the job cancelation message, transmit an indication of cancelation of the job flow to a requesting device.

Automated message-based job flow resource management in container-supported many task computing

An apparatus includes processor(s) to: receive, a request to cancel a performance of a job flow; store, within a task kill queue, a kill tasks request message; within a task container, in response to the storage of the kill tasks request message, cease execution of instructions of a task routine to cease performance of a task of a set of tasks of the job flow, and store, within a task queue, a task cancelation message indicating cancelation of execution of the task routine; within a performance container, execute instructions of a performance routine to cause the at least one processor to, in response to the storage of the task canceled message, store, within a job queue, a job cancelation message indicating cancelation of the job flow; and in response to the storage of the job cancelation message, transmit an indication of cancelation of the job flow to the requesting device.

Commanded message-based job flow cancellation in container-supported many task computing

An apparatus includes a processor to: receive a request to perform a set of tasks; analyze input data objects used as inputs and output data objects generated as outputs by the set of tasks to identify at least one dependency between tasks and at least one pair of tasks performable in parallel; determine an order of performance of the set of tasks; retrieve executable instructions of a corresponding set of task routines that define input interfaces and output interfaces for the input data objects and output data objects, respectively; generate a job flow definition defining the order of performance, each input interface and each output interface; use the job flow definition to perform the set of tasks; and transmit one of the output data objects to the requesting device as a result report serving as the output of the set of tasks.

Many task computing with message passing interface

An apparatus includes a processor to: receive a request from a remote device to perform a job flow; retrieve a job flow definition defining the job flow and each of a set of task routines to perform tasks of the job flow from a set of storage devices where each is stored as an undivided object within one storage device; and in response to determining that a data set is stored as multiple data object blocks, generate a container containing the job flow definition and set of task routines to enable each storage device to perform the job flow using a locally stored data object block of the data set as input to generate a corresponding data object block of a result report, provide a copy of the container to each storage device, and transmit the result report assembled from the data object blocks thereof to the remote device.

Growing community of inventors

Morrisville, NC, United States of America

Eric Jian Yang