International Business Machines Corporation

Globalfoundries Inc.

Vincent Kulandaisamy

Vijayshankar Raman

Ronald Jason Barber

Liping Zhang

Gopi K Attaluri

Sam Sampson Lightstone

Ippokratis Pandis

Guy Maring Lohman

Richard Sefton Sidle

Ruchir Puri

Rajesh Ramkrishna Bordawekar

Minsik Cho

Naresh K Chainani

Ulrich Alfons Finkler

Daniel Brand

Calisto Paul Zuzarte

Oded Shmueli

Brian R Konigsburg

Steven Robert Pearson

Wenbin Ma

Dale Murray McInnis

Jon A Lind

Yuke Zhuge

Wayne Jonleon Young

Steve Raspudic

Gregory R Stager

Joshua D Gross

Embodiments of the invention relate to processing data records, and for a multi-phase partitioned data reduction. The first phase relates to processing data records and partitioning the records into a first partition of records having a common characteristic and a second partition of records that are not members of the first partition. The data records in each partition are subject to intra-partition data reduction responsive to a resource constraint. The data records in each partition are also subject to an inter-partition data reduction, also referred to as an aggregation to reduce a footprint for storing the records. Partitions and/or individual records are logically aggregated and a data reduction operation for the logical aggregation of records takes place in response to available resources.

Accurate partition sizing for memory efficient reduction operations

A database system, computer program product, and a method for evaluating aggregates in database systems includes hashing of aggregation keys on a per bucket basis, and depending on a number of hashed tuples per bucket, sorting said tuples. Additionally, depending on the number of hashed tuples per bucket, the bucket is kept without change. Moreover, depending on the number of hashed tuples per bucket, maintaining a secondary hash table for a particular bucket, map tuples to it, aggregate as you map.

Method and system for hybrid sort and hash-based query execution

Provided herein are techniques for processing a database query aggregating data. Data tuples of a database object each including a grouping element and a data element are analyzed to determine a length of the data element for each data tuple. A plurality of tables each accommodate a successively greater length for the data element. A corresponding table of the plurality of tables to store each data tuple is determined based on the length of the data element of that data tuple relative to the accommodated lengths of the plurality of tables. Each data tuple in the determined corresponding table is stored to group the data tuples within each of the plurality of tables based on the grouping element, and an indication of corresponding tables containing members for each group is provided. The groups are combined across the plurality of tables to aggregate the data tuples for a database query.

Efficient database query aggregation of variable length data

A computer program product provides efficient multiple aggregation distinct processing. The computer program product including a computer readable storage medium having program instructions embodied therewith. The program instructions are executable by a processor to cause the processor to convert a query with multiple aggregation distincts into a two-stage group-by process using a nest operator. The two-stage group-by process including further program instructions to cause the processor to: perform a first stage group-by process including the nest operator taking a single input row, and concatenating a grouping key with a measure value for each aggregation distinct that forms multiple inputs to a single group-by table, and perform a second stage group-by process including the nest operator bringing together entries for each original group.

Efficient multiple aggregation distinct processing

Embodiments include a method, system, and computer program product for encoding data while it is being processed as part of a query is provided. The method includes receiving a query request and determining a set of values associated with data to be encoded for completing the query request. The method also includes encoding those values such that any subsequent processing operations can be performed on the encoded values to complete the requested query. After performing the subsequent processing operations to complete the requested query, each value is decoded back to its original value.

On-the-fly encoding method for efficient grouping and aggregation

A method, a system, and a computer program product are provided. A filter is created for each portion of a data set. The filter indicates which one or more characteristics are present among each of the portions. Each of the one or more characteristics comprises one or more groups defined by the data grouping operation. The filters for the portions of the data sets are transferred to one or more filter processors and combined within the one or more filter processors to indicate characteristics of data residing across multiple processing elements to produce a result for a data grouping operation, utilizing transfers based on a combined filter result. In various embodiments, the filter may be a Bloom filter.

Efficient aggregation in a parallel system

A hardware accelerator includes a bitmap processor that processes a bitmap structure for multiple list population. A population count processor processes population counts for data and aggregates the population counts. The bitmap data structure includes bitmap bit fields interleaved with aggregated population count fields.

Hardware for a bitmap data structure for efficient storage of heterogeneous lists

Systems and methods for sorting a data set stored on an external device. A plurality of smaller radix sizes are determined, based on a first radix size and performance characteristics of an external data storage device, whose sizes add up to a first radix size for an in-place radix sort. The smaller radix sizes reduce a total time to perform the in-place radix sort. Each level of a multiple level in-place radix sort is performed with the smaller radix sizes. Each level of the sort includes dividing the data set into N buckets; dividing the buffer into N buckets; and iteratively loading a respective segment in each bucket of the data set into a respective bucket of the buffer, performing an in-place radix sort on the data in the buffer, and returning sorted buffer data to the data set on the external storage device.

Adaptive radix external in-place radix sort

Methods for sorting a data set. Data items each having a first portion and a second portion is stored. The first and second portions are stored separately and each has a separate set of keys. The first portion has a pointer indicating the second portion. At least some of the first set of keys for each data item is stored in a local memory of a first processor. At least one data stripe set is defined with one stripe within each bucket. An in-place partial bucket radix sort is performed on data items within one data stripe set with a first processor using an initial key. Incorrectly sorted data items are grouped into respective incorrect data item groups within each bucket. A radix sort is then performed using the initial radix on the incorrect data item groups. A first level sorted output is produced.

Parallelized in-place radix sorting

Systems and methods for sorting a data set. Data items each having a first portion and a second portion is stored. The first and second portions are stored separately and each has a separate set of keys. The first portion has a pointer indicating the second portion. At least some of the first set of keys for each data item is stored in a local memory of a first processor. At least one data stripe set is defined with one stripe within each bucket. An in-place partial bucket radix sort is performed on data items within one data stripe set with a first processor using an initial key. Incorrectly sorted data items are grouped into respective incorrect data item groups within each bucket. A radix sort is then performed using the initial radix on the incorrect data item groups. A first level sorted output is produced.

Embodiments include a method and computer program product for encoding data while it is being processed as part of a query is provided. The method includes receiving a query request and determining a set of values associated with data to be encoded for completing the query request. The method also includes encoding those values such that any subsequent processing operations can be performed on the encoded values to complete the requested query. After performing the subsequent processing operations to complete the requested query, each value is decoded back to its original value.

Embodiments include a system for encoding data while it is being processed. The system includes a processor, an encoder and a decoder. The processor is configured to process a query request by determining a set of values. The encoder is configured for encoding the set of values, such that a subsequent processing operation can be performed on the encoded values. The processor performs the subsequent processing operations. The decoder is configured for decoding each value back to its value prior to being encoded upon completion of the processor completing the requested query.

Embodiments include method, system, and computer program product for providing aggregation hierarchy that is related memory hierarchies. In one embodiment, the method includes determining capacity of a first level memory of a memory hierarchy for processing data relating to completion of an aggregation process and generating a per thread local look-up table in said first level memory upon determining said capacity. Upon the first level memory reaching capacity, a plurality of per thread partitions to store remaining data to complete the aggregation process in a second level memory of the memory hierarchy is generated such that each of said per-thread partitions includes an identical amount of data portion on each thread. The method also includes storing the per thread partitions in said second level memory and providing a single global look up table for each of the identical data portions.

Multi-level aggregation techniques for memory hierarchies

A method, system, and computer program product for reducing likelihood of data loss during performance of failovers in a high-availability system comprising a primary system and a standby system are provided. The method, system, and computer program product provide for defining a halt duration, periodically determining a halt end time, halting data modifications at the primary system responsive to failure of data replication to the standby system, resuming data modifications at the primary system responsive to a last determined halt end time being reached or data replication to the standby system resuming, and responsive to the primary system failing prior to a previously determined halt end time, determining that a failover to the standby system will not result in data loss on the standby system with respect to the primary system.

Growing community of inventors

Portland, OR, United States of America

Vincent Kulandaisamy