International Business Machines Corporation

Green Market Square Limited

Austin Clifford

Konrad Emanowicz

Enda McCallig

Pedro Miguel Barbas

Gary Murtagh

Clare Scally

Ilker Ender

Sami Abed

Gareth Jenkins

Patrick G O'Sullivan

Mara Elisa De Paiva Fernandes Matias

Hemant Asandas Bhatia

Enzo Cialini

Garrett Fitzsimons

Mara Matias

Brian McKeown

A computer-implemented method, system and computer program product for optimally performing stress testing against big data management systems. A set of random test queries is generated and compiled to determine the data points of the features (e.g., table type being queried) of the set of random test queries. A distance (e.g., Mahalanobis distance) is then measured between the data points of the features and the mean of a distribution of data points corresponding to each same feature of an extracted feature set. Each random test query whose distance exceeds a threshold distance is then ranked. The ranked random test queries are then executed in order of rank. Those executed random test queries which resulted in an error (e.g., system failure) are added to a log, which is used to identify those queries to perform a stress test against the big data management system.

Workload generation for optimal stress testing of big data management systems

In an approach for automatic vertical partitioning of fact tables in a distributed query engine a processor analyzes a sample end-user workload of queries to extract filter predicates associated with each of multiple fact tables relating to a big data store. A processor, for each fact table, and for each column in the fact table to which a filter predicate is applied and where coarsification is required, generates a candidate partitioning expression incorporating an adjustment to a coarsification function based on a data distribution of values in the column. A processor scores the candidate partitioning expressions for each fact table based on cost data relating to the sample end-user workload and selects one or more candidate partitioning expressions to optimize partitioning of each fact table with each partition data being placed in a separate directory in a distributed file system.

Automatic vertical partitioning of fact tables in a distributed query engine

Aspects of the present invention disclose a method, computer program product, and system for auto-scaling a query engine. The method includes one or more processors monitoring query traffic at the query engine. The method further includes one or more processors classifying queries by a plurality of service classes based on a level of complexity of a query. The method further includes one or more processors comparing query traffic for each service class with a concurrency threshold of a maximum number of queries of the service class allowed to be concurrently processed. The method further includes one or more processors instructing auto-scaling of a cluster of worker nodes to change a number of worker nodes available in the cluster based on the comparison, over a defined period of time, of the query traffic relative to a defined upscaling threshold and a defined downscaling threshold.

Auto-scaling a query engine for enterprise-level big data workloads

A computer-implemented method, computer program product and system for identifying pseudonymized data within data sources. One or more data repositories within one or more of the data sources are selected. One or more privacy data models are provided, where each of the privacy data models includes pattern(s) and/or parameter(s). One or more of the one or more privacy data models are selected. Data identification information is generated, where the data identification information indicates a presence or absence of pseudonymized data and of non-pseudonymized data within the one or more of the data sources. The data identification information is generated utilizing the pattern(s) and/or the parameter(s) to determine pseudonymized data.

Runtime control of automation accuracy using adjustable thresholds

Proposed are concepts for providing resilience (i.e., fault tolerance) for the temporary data needs of a distributed file system. Such concepts may, for instance, provide a virtual storage layer in a data node of a distributed file system. The virtual storage layer may provide resilience for the temporary data needs of a Massively Parallel Processing (MPP) SQL on Hadoop engine.

Temporary data storage in data node of distributed file system

Provided are a system, method and computer program product for redistribution of data in an online shared nothing database, said shared nothing database comprising a plurality of original partitions and at least one new partition. Said method comprising: a collocation-set identifier for identifying two or more collocated tables in the original partitions to be redistributed over the original partitions and at least one new partition, said tables having data rows, each data row residing on a partition governed by a distribution key; a table factory for creating corresponding new tables in the new partition; a view creator for creating a UNION ALL view of all the data from both the original and new tables using the check constraints, said check constraints in turn based on the distribution keys; a counter for defining a loop of intervals over the distribution key range; a data mover for performing data movement of each data row over the current distribution key interval from the original tables to the new tables; a data changer for altering the check constraints for each distribution key interval so that the UNION ALL view is preserved thereby ensuring collocation and data integrity is maintained; a loop finisher for looping back for the next distribution key interval until all the distribution key intervals have been processed such that all the data has been moved to the new table whereby collocation and data integrity is maintained, redistribution is transparent for queries performed on the database, and tables are fully available during redistribution for read and write.

Incremental and collocated redistribution for expansion of online shared nothing database

Embodiments of the present invention provide a method, system and computer program product for test data generation using unique common factor sequencing. In an embodiment of the invention, a method for test data generation using unique common factor sequencing is provided. The method includes loading a table for population with test data in a test data generation tool executing in memory of a computer. A column set of multiple columns in the table associated with a key to the table can be selected for processing and different cardinality sequence values are assigned to the columns in the set such that the cardinality sequence values do not share a common factor except for unity as in the case of prime numbers.

Test data generation and scale up for database testing using unique common factor sequencing

Disclosed is an approach comprising a column partitioned into a plurality of partitions including an empty partition and a plurality of filled partitions each comprising data entries associated with a set of parameters having parameter values, the data entries compressed in accordance with a compression dictionary. The approach comprises receiving forecasted parameter values for an expected set of data entries to be stored in an empty partition; predicting a recurrence frequency of the data entries in the expected set using the forecasted parameter values by evaluating the respective compression dictionaries of the filled partitions with a machine learning algorithm; generating a predictive compression dictionary for the expected set of data entries based on the predicted recurrence frequency of the data entries in the expected set; receiving the expected set of data entries; and compressing at least part of the received expected set of data entries using the predictive compression dictionary.

Columnar database compression

Identification of pseudonymized data within data sources

Embodiments of the present invention provide a method, system and computer program product for test data generation using unique common factor sequencing. In an embodiment of the invention, a method for test data generation using unique common factor sequencing is provided. The method includes loading a table for population with test data in a test data generation tool executing in a memory of a computer. A column set of multiple columns in the table associated with a key to the table is selected for processing and different cardinality sequence values are assigned to the columns in the set such that the cardinality sequence values do not share a common factor except for unity as in the case of prime numbers.

Disclosed is a computer-implemented method of compressing data in a columnar database comprising at least one column partitioned into a plurality of partitions including at least one empty partition and a plurality of filled partitions each comprising data entries associated with a set of parameters having parameter values relevant to the recurrence frequency of the data entry in the partition, the data entries being compressed in accordance with a compression dictionary based on the respective recurrence frequencies of the data entries in the filled partition. The computer-implemented method comprises receiving forecasted parameter values for the set of parameters for an expected set of data entries to be stored in an empty partition of the column; predicting a recurrence frequency of the data entries in the expected set using the forecasted parameter values by evaluating the respective compression dictionaries of the filled partitions with a machine learning algorithm; generating a predictive compression dictionary for the expected set of data entries based on the predicted recurrence frequency of the data entries in the expected set; receiving the expected set of data entries; and compressing at least part of the received expected set of data entries using the predictive compression dictionary. A computer program product and a computer system for implementing such a method are also disclosed.

In an exemplary embodiment of this disclosure, a method for loading data from a backup image of a database includes selecting a subset statement defining a subset of the data in the database. Tables of the database are identified based on metadata of the database. A target database is written having the structure but not the data of the identified tables. One or more table statements are constructed, by a computer processor, defining a subset of each identified table based on the subset statement. Selected data is unloaded from a backup image into the target database using respective table statements as filters.

Growing community of inventors

Dublin, Ireland

Austin Clifford