Emc IP Holding Company LLC

Yania Molina Souto

Fábio André Machado Porto

Vinícius Michel Gottin

Jonas Furtado Dias

Angelo E M Ciarlini

Paulo De Figueiredo Pires

Edward José Pacheco Condori

Wagner Dos Santos Vieira

Bruno Carlos Da Cunha Costa

Vinicius Michel Gottin

Techniques are provided for dataflow execution time estimation for distributed processing frameworks. An exemplary method comprises: obtaining an input dataset for a dataflow for execution; determining a substantially minimal data unit for a given operation of the dataflow processed by the given operation; estimating a number of rounds required to execute a number of data units in the input dataset using nodes assigned to execute the given operation; determining an execution time spent by the given operation to process one data unit; estimating the execution time for the given operation based on the execution time spent by the given operation to process one data unit and the number of rounds required to execute the number of data units in the input dataset; and executing the given operation with the input dataset. A persistent cost model is optionally employed to record the execution times of known dataflow operations.

Dataflow execution time estimation for in-memory distributed processing framework

A dataflow execution environment is provided with dynamic placement of cache operations. An exemplary method comprises: obtaining a first cache placement plan for a dataflow comprised of multiple operations; executing operations of the dataflow and updating a number of references to the executed operations to reflect remaining executions of the executed operations; determining a current cache gain by updating an estimated reduction in the total execution cost for the dataflow of the first cache placement plan; determining an alternative cache placement plan for the dataflow following the execution; and implementing the alternative cache placement plan based on a predefined threshold criteria. A cost model is optionally updated for the executed operations using an actual execution time of the executed operations. A cached dataset can be removed from memory based on the number of references to the operations that generated the cached datasets.

In-memory dataflow execution with dynamic placement of cache operations

A dataflow execution environment is provided with dynamic placement of cache operations and action execution ordering. An exemplary method comprises: obtaining a current cache placement plan for a dataflow comprised of multiple operations and a corresponding current cache gain estimate; selecting an action to execute from a plurality of remaining dataflow actions based on a predefined policy; executing one or more operations in a lineage of the selected action and estimating an error as a difference in an observed execution time and an estimated execution time given by a cost model; obtaining an alternative cache placement plan for the dataflow following the execution in conjunction with a predefined new plan determination criteria being satisfied and a corresponding alternative cache gain estimate; implementing the alternative cache placement plan in conjunction with a predefined new plan implementation criteria being satisfied; and selecting a next action to execute from a plurality of remaining actions in the dataflow based on a predefined policy.

In-memory dataflow execution with dynamic placement of cache operations and action execution ordering

Techniques are provided for automatic placement of cache operations in a dataflow. An exemplary method obtains a graph representation of a dataflow of operations; determines a number of executions and a computational cost of the operations, and a computational cost of a caching operation to cache a dataset generated by an operation; establishes a dataflow state structure recording values for properties of the dataflow operations for a number of variations of caching various dataflow operations; determines a cache gain factor for dataflow operations as an estimated reduction in the accumulated cost of the dataflow by caching an output dataset of a given operation; determines changes in the dataflow state structure by caching an output dataset of a different operation in the dataflow; and searches the dataflow state structures to determine the output datasets to cache based on a total dataflow execution cost.

Growing community of inventors

Petrópolis, Brazil

Yania Molina Souto