Emc IP Holding Company LLC

Emc Corporation

Angelo E M Ciarlini

Rômulo Teixeira De Abreu Pinho

Alex Laier Bordignon

Jonas Furtado Dias

Edward José Pacheco Condori

Vinícius Michel Gottin

André De Almeida Maximo

Vinicius Michel Gottin

Adriana Bechara Prado

Paulo De Figueiredo Pires

Wagner Dos Santos Vieira

Bruno Carlos Da Cunha Costa

Marcello Luiz Rodrigues De Campos

Jaumir Valença Da Silveira Junior

Percy E Rivera Salas

Fábio André Machado Porto

Yania Molina Souto

Monica Barros

Fabio A M Porto

Daniel Sadoc Menasché

Rodrigo Dias Arruda Senra

Victor Bursztyn

Fábio A Perosi

Rodrigo De Souza Lima Espinha

Mauricio Melo Camara

Renato M M Medeiros

Silvana Rossetto

Timothy A Voyt

Noel Moreno Lemus

Luiz Guilherme Oliveira Dos Santos

Jonas F Bias

Amir H K Moghadam

Fábio A M Porto

Thiago De Sa Feital

Rafael Marinho Soares

José Carlos Costa Da Silva Pinto

André Maximo

Methods and apparatus are provided for evaluating combinatorial processes using simulation techniques and multiple parallel statistical analyses of real-world data. A simulation model is generated that simulates one or more steps of a combinatorial process. The simulation model comprises key features of the combinatorial process. A plurality of first data mining tasks are performed in parallel over real data of the combinatorial process to obtain key feature prediction models that estimate the key features. The key feature prediction models are bound to the simulation model. Query types to be supported are identified and a plurality of simulation runs are generated in parallel, comprising simulated data for the supported query types. A plurality of second data mining tasks are performed in parallel over the plurality of simulation runs to build global prediction models to answer queries of each supported query type. An answer to a user query is determined using the global prediction models.

Methods and apparatus for evaluation of combinatorial processes using simulation and multiple parallel statistical analyses of real data

Methods and apparatus are provided for process monitoring based on large-scale combinations of time series data. An exemplary method comprises generating a model from time series data for a given target time series; determining whether a first difference between measured values and predicted values based on the model exceeds a predefined threshold indicating a target prediction error; in response to a detected target prediction error, performing evaluations of (i) a neighborhood coherence comprising an average of variables of the model weighted by corresponding coefficients on a predefined neighborhood time window, and/or (ii) a second difference between a given value of at least one variable in the model and an average value of the at least one variable based on a training dataset; providing notifications when first predefined criteria based on the evaluations are satisfied; and updating the model when second predefined criteria based on the evaluations are satisfied.

Process monitoring based on large-scale combination of time series data

Product planning techniques are provided that recommend compositions of product features for weighted heterogeneous consumer segments using regression trees. An exemplary method comprises obtaining historical consumer data comprising product preferences for existing product items for multiple consumer segments; obtaining product features indicating characteristics for each existing product item; prioritizing the consumer segments by obtaining a weight indicating an interest in each consumer segment; computing a total performance metric, for each product item, by calculating a dot product between the consumer segment weights and respective preferences of the consumer segments regarding a given product item; obtaining a regression tree from the existing product items to predict the total performance metric in terms of corresponding product features; and selecting a combination of the product features to be used in future product items based on identified paths in the regression tree.

Method, medium, and system for recommending compositions of product features using regression trees

Techniques are provided for simulation-based online workflow optimization. One method comprises obtaining a state of concurrent workflows; obtaining state similarity functions that assign a similarity score between pairs of states; generating a simulation model of the workflow executions representing different configurations of at least one control variable in the workflow executions by mapping states with a highest similarity; obtaining at least one utility function that assigns a utility score to the states in the simulation model; determining a configuration of the control variable that maximizes the utility score for states in the simulation model; and for new concurrent workflows: obtaining a current state of the new concurrent workflows; identifying a most similar state with one of the determined configurations of the control variable in the simulation model with a highest similarity to the current state; and adjusting the configuration of the control variable of the new concurrent workflows to match the corresponding configuration of the control variable of the most similar state.

Simulation-based online optimization of workflows

Methods and apparatus are provided for taking snapshots to train prediction models and improve workflow execution. An exemplary method for managing execution of a workflow comprising one or more sub-workflows comprises parsing and interpreting a definition of the workflow and the one or more sub-workflows; flattening the workflow and the sub-workflows into a directed acyclic graph where nodes represent activities and edges represent partial-order constraints; collecting provenance data about executions of the workflow according to the directed acyclic graph; taking snapshots of the collected provenance data as a representation of a state of the workflow execution; calculating one or more quality metrics regarding the execution of the workflow; associating the snapshots to the quality metrics; training one or more prediction models of the quality metrics based on the snapshots of the workflow executions; and processing the prediction models during a real-time execution of the workflow to adjust the execution of the workflow. The snapshots are triggered using graph-based and/or time-based triggering techniques.

Snapshots to train prediction models and improve workflow execution

Techniques are provided for dataflow execution time estimation for distributed processing frameworks. An exemplary method comprises: obtaining an input dataset for a dataflow for execution; determining a substantially minimal data unit for a given operation of the dataflow processed by the given operation; estimating a number of rounds required to execute a number of data units in the input dataset using nodes assigned to execute the given operation; determining an execution time spent by the given operation to process one data unit; estimating the execution time for the given operation based on the execution time spent by the given operation to process one data unit and the number of rounds required to execute the number of data units in the input dataset; and executing the given operation with the input dataset. A persistent cost model is optionally employed to record the execution times of known dataflow operations.

Dataflow execution time estimation for in-memory distributed processing framework

Simulation data is summarized and queried. A user provides an indication of simulation data that will be subsequently queried. The queried simulation data comprises (i) a set of key attributes, (ii) a set of events, and/or (iii) a set of causality relationships between a plurality of the events. First level summaries summarize simulation executions of scenarios of a combinatorial process and comprise (i) a summary of the frequency distribution of key attribute values, (ii) a timestamp for each event, and (iii) an indication of causality between events observed during the simulation. Second level summaries summarize executions of the given scenario and comprise (i) a consolidated distribution probability for the key attributes, (ii) a frequency distribution of occurrences of the events in a single execution, and (iii) a frequency of observations of the causality between pairs of events. In response to a query, second level summaries of selected scenarios are accessed to retrieve information related to the elements expressed in the query and to produce a third level summary that aggregates information accessed from the second level summaries of the selected scenarios.

Summarizing and querying data generated from multiple scenarios of a data-intensive simulation

Methods and apparatus are provided for data operation pre-processing with a probabilistic estimation of operation value. An exemplary method comprises extracting feature values from a data set; identifying a set of operations that previously processed data sets comprising the extracted feature values; determining whether to execute an operation from the set of operations before an explicit request for the execution of the operation based on a probabilistic evaluation of a value of pre-processing the operation; and executing a set of instructions for the operation when it is determined that the operation is to be executed. The set of instructions comprises, e.g., stopping execution of other operations being executed, freeing resources required by the operation, and/or allocating resources required by the operation. An optional indirect component, optionally scaled by a moderating factor, estimates the value of pre-processing one or more additional operations enabled by features of data sets produced by the operation, and the resources to be allocated for the pre-processing of those operations.

Methods and apparatus for data operation pre-processing with probabilistic estimation of operation value

Techniques are provided for automatic combination of sub-process simulation results with dataset selection based on a fitness under one or more specific scenarios. An exemplary method comprises obtaining an execution map for each sub-process in a sequence that stores results of a given sub-process execution. The results comprise a scenario, a distribution and a distribution fitness value. In response to a user query regarding a target feature and an initial dataset, initial dataset are combined with results selected from the execution map for a first sub-process in the sequence; each available dataset from the previous sub-processes in the sequence is combined with results selected from the execution map for the next sub-process; a probability distribution function (pdf) for the target feature is composed from a combined dataset that represents a simulation of the process and combines results of each of sub-process in the sequence; and the pdf is processed to answer the user query for the target feature.

Automatic combination of sub-process simulation results with dataset selection based on fitness under specific scenarios

Methods and apparatus are provided for automatic combination of sub-process simulation results and heterogeneous data sources. An exemplary method comprises obtaining, for a process comprised of a sequence of a plurality of sub-processes, an identification of relevant input and output features for each sub-process; obtaining an execution map for each sub-process, wherein each execution map stores results of an execution of a given sub-process; and, in response to a user query regarding a target feature and a user-provided initial scenario: composing a probability distribution function for the target feature representing a simulation of the process based on a sequence of the execution maps, by matching input features of each execution map with features from the initial scenario or the output of previous execution maps; and processing the probability distribution function to answer the user query. Execution maps are optionally stored as distributed tables that use relevant input features to hash data related to multiple executions across multiple nodes. The composition process optionally occurs in parallel across multiple nodes.

Automatic combination of sub-process simulation results and heterogeneous data sources

Pattern queries are evaluated in parallel over large N-dimensional datasets to identify features of interest. Similarity-based pattern matching tasks are executed over N-dimensional input datasets comprised of numeric values by providing data representations for the N-dimensional input datasets, a pattern query and one or more candidate solutions for the pattern query, such that the pattern query specifies a pattern of an N-dimensional body that is compared to at least one candidate solution corresponding to an N-dimensional body extracted from the N-dimensional input datasets; defining a distance metric that compares the N-dimensional body formed by the candidate solution extracted from the N-dimensional input datasets and the N-dimensional body formed by the pattern query, taking into account one or more of the following criteria: differences between mapped values, differences in scale and differences in shape; and executing, in parallel, a plurality of independent instances of at least one algebraic operator to generate and score the candidate solutions based on the distance metric. The exemplary algebraic operators comprise a Trace Match operator, a Ranking operator, a Candidate Solution operator, and a Query Clustering operator.

Methods and apparatus for parallel evaluation of pattern queries over large N-dimensional datasets to identify features of interest

Methods and apparatus are provided for performing massively parallel processing (MPP) large-scale combinations of time series data. A given working compute node in a distributed computing environment obtains a given group of time series data of a plurality of groups of time series data; generates a measurement matrix for the given group based on a plurality of selected time series and a plurality of time lags of the selected time series; processes the measurement matrix to generate a first linear model with a predefined number of first independent selected variables; assigns a score to each first independent selected variable; and provides the first independent selected variables and assigned scores to a master compute node that ranks the first independent selected variables for all groups from all working computing nodes according to assigned scores; selects a predefined number of second independent selected variables based on a final rank to create a final group of time series; and processes the final group of time series to generate a final linear model.

Massively parallel processing (MPP) large-scale combination of time series data

Techniques are provided for automatic placement of cache operations in a dataflow. An exemplary method obtains a graph representation of a dataflow of operations; determines a number of executions and a computational cost of the operations, and a computational cost of a caching operation to cache a dataset generated by an operation; establishes a dataflow state structure recording values for properties of the dataflow operations for a number of variations of caching various dataflow operations; determines a cache gain factor for dataflow operations as an estimated reduction in the accumulated cost of the dataflow by caching an output dataset of a given operation; determines changes in the dataflow state structure by caching an output dataset of a different operation in the dataflow; and searches the dataflow state structures to determine the output datasets to cache based on a total dataflow execution cost.

Automatic placement of cache operations for complex in-memory dataflows

Lossless content-aware compression and decompression techniques are provided for floating point data, such as seismic data. A minimum-length compression technique exploits an association between an exponent and a length of the significand, which corresponds to the position of the least significant bit of the significand. A reduced number of bits from the significand can then be stored. A prediction method is also optionally previously applied, so that residual values with shorter lengths are compressed instead of the original values. An alignment compression technique exploits repetition patterns in the floating point numbers when they are aligned to the same exponent. Floating point numbers are then split into integral and fractional parts. The fractional part is separately encoded using a dictionary-based compression method, while the integral part is compressed using a delta-encoding method. A prediction method is also optionally previously applied to the integral part, thereby increasing the compression ratio.

Content-aware lossless compression and decompression of floating point data

Distributed content-aware compression and decompression techniques are provided for data, such as seismic data. Data is compressed by obtaining a file chunk of a plurality of file chunks of a larger file, a start offset of the file chunk, and a data descriptor indicating how the file chunk should be processed based on characteristics of the larger file. Headers in the file chunk are compressed using a substantially lossless header compression technique to obtain compressed headers. Samples in the file chunk are compressed using a substantially lossless sample compression technique to obtain compressed samples. The compressed headers and compressed samples are packed into a single bit stream comprising a compressed version of the file chunk. The compression can be performed in a physical or logical layer of storage nodes of a file system or in compute nodes of a computing cluster. The compression can be executed on demand by an external agent and/or in a background mode by a storage operating system.

Growing community of inventors

Rio de Janeiro, Brazil

Angelo E M Ciarlini