International Business Machines Corporation

Ilker Ender

Austin Clifford

Mara Elisa De Paiva Fernandes Matias

Hemant Asandas Bhatia

Pedro Miguel Barbas

Mara Matias

A computer-implemented method, system and computer program product for optimally performing stress testing against big data management systems. A set of random test queries is generated and compiled to determine the data points of the features (e.g., table type being queried) of the set of random test queries. A distance (e.g., Mahalanobis distance) is then measured between the data points of the features and the mean of a distribution of data points corresponding to each same feature of an extracted feature set. Each random test query whose distance exceeds a threshold distance is then ranked. The ranked random test queries are then executed in order of rank. Those executed random test queries which resulted in an error (e.g., system failure) are added to a log, which is used to identify those queries to perform a stress test against the big data management system.

Workload generation for optimal stress testing of big data management systems

In an approach for automatic vertical partitioning of fact tables in a distributed query engine a processor analyzes a sample end-user workload of queries to extract filter predicates associated with each of multiple fact tables relating to a big data store. A processor, for each fact table, and for each column in the fact table to which a filter predicate is applied and where coarsification is required, generates a candidate partitioning expression incorporating an adjustment to a coarsification function based on a data distribution of values in the column. A processor scores the candidate partitioning expressions for each fact table based on cost data relating to the sample end-user workload and selects one or more candidate partitioning expressions to optimize partitioning of each fact table with each partition data being placed in a separate directory in a distributed file system.

Automatic vertical partitioning of fact tables in a distributed query engine

Aspects of the present invention disclose a method, computer program product, and system for auto-scaling a query engine. The method includes one or more processors monitoring query traffic at the query engine. The method further includes one or more processors classifying queries by a plurality of service classes based on a level of complexity of a query. The method further includes one or more processors comparing query traffic for each service class with a concurrency threshold of a maximum number of queries of the service class allowed to be concurrently processed. The method further includes one or more processors instructing auto-scaling of a cluster of worker nodes to change a number of worker nodes available in the cluster based on the comparison, over a defined period of time, of the query traffic relative to a defined upscaling threshold and a defined downscaling threshold.

Auto-scaling a query engine for enterprise-level big data workloads

Proposed are concepts for providing resilience (i.e., fault tolerance) for the temporary data needs of a distributed file system. Such concepts may, for instance, provide a virtual storage layer in a data node of a distributed file system. The virtual storage layer may provide resilience for the temporary data needs of a Massively Parallel Processing (MPP) SQL on Hadoop engine.

Growing community of inventors

Dublin, Ireland

Ilker Ender