Emc IP Holding Company LLC

Emc Corporation

Bruno Carlos Da Cunha Costa

Angelo E M Ciarlini

Paulo De Figueiredo Pires

Wagner Dos Santos Vieira

Jonas Furtado Dias

Edward José Pacheco Condori

Fábio André Machado Porto

Yania Molina Souto

Vinicius Michel Gottin

Vinícius Michel Gottin

Alex Laier Bordignon

Fabio A M Porto

Fábio A Perosi

Amir H K Moghadam

Jonas F Bias

Techniques are provided for dataflow execution time estimation for distributed processing frameworks. An exemplary method comprises: obtaining an input dataset for a dataflow for execution; determining a substantially minimal data unit for a given operation of the dataflow processed by the given operation; estimating a number of rounds required to execute a number of data units in the input dataset using nodes assigned to execute the given operation; determining an execution time spent by the given operation to process one data unit; estimating the execution time for the given operation based on the execution time spent by the given operation to process one data unit and the number of rounds required to execute the number of data units in the input dataset; and executing the given operation with the input dataset. A persistent cost model is optionally employed to record the execution times of known dataflow operations.

Dataflow execution time estimation for in-memory distributed processing framework

Pattern queries are evaluated in parallel over large N-dimensional datasets to identify features of interest. Similarity-based pattern matching tasks are executed over N-dimensional input datasets comprised of numeric values by providing data representations for the N-dimensional input datasets, a pattern query and one or more candidate solutions for the pattern query, such that the pattern query specifies a pattern of an N-dimensional body that is compared to at least one candidate solution corresponding to an N-dimensional body extracted from the N-dimensional input datasets; defining a distance metric that compares the N-dimensional body formed by the candidate solution extracted from the N-dimensional input datasets and the N-dimensional body formed by the pattern query, taking into account one or more of the following criteria: differences between mapped values, differences in scale and differences in shape; and executing, in parallel, a plurality of independent instances of at least one algebraic operator to generate and score the candidate solutions based on the distance metric. The exemplary algebraic operators comprise a Trace Match operator, a Ranking operator, a Candidate Solution operator, and a Query Clustering operator.

Methods and apparatus for parallel evaluation of pattern queries over large N-dimensional datasets to identify features of interest

Techniques are provided for automatic placement of cache operations in a dataflow. An exemplary method obtains a graph representation of a dataflow of operations; determines a number of executions and a computational cost of the operations, and a computational cost of a caching operation to cache a dataset generated by an operation; establishes a dataflow state structure recording values for properties of the dataflow operations for a number of variations of caching various dataflow operations; determines a cache gain factor for dataflow operations as an estimated reduction in the accumulated cost of the dataflow by caching an output dataset of a given operation; determines changes in the dataflow state structure by caching an output dataset of a different operation in the dataflow; and searches the dataflow state structures to determine the output datasets to cache based on a total dataflow execution cost.

Automatic placement of cache operations for complex in-memory dataflows

Growing community of inventors

Teresópolis, Brazil

Bruno Carlos Da Cunha Costa