Hash-based database grouping system and method

Abstract

A structured query language (SQL) grouping and aggregation system and method that incorporates hash-based techniques, several overflow handling strategies and statistics-based process-selection criteria. The method can execute SQL group-by queries on distributed database tables or tables stored locally to the database management system (DBMS) processor executing the grouping method. Hash-based techniques allow groupings and aggregates to be generated on the fly through the use of partial aggregates maintained in primary memory. Where primary memory is limited, groups and aggregates are still generated for as many groups as can be maintained in primary memory, while various overflow procedures are provided for buffering ungrouped data and writing that data to an overflow disk file for later processing. In one overflow procedure, raw data from groups that cannot be aggregated in primary memory are buffered then written to the overflow disk file. In a second overflow procedure, ungroupable raw data is formatted the same as data being aggregated in the group table, buffered, and then written to the overflow file. In a third overflow procedure, ungroupable raw data is partially aggregated in an output buffer maintained in primary memory before being written to the overflow file maintained in secondary memory. Database table statistics maintained by a cataloger are consulted to determine whether hash-based grouping or conventional sort based grouping should be used to execute a group-by query. The system is adaptable to running a grouping query against a partitioned database on distributed processors.