International Business Machines Corporation

Globalfoundries Inc.

Ippokratis Pandis

Vijayshankar Raman

Guy Maring Lohman

Ronald Jason Barber

Gopi K Attaluri

Rene Mueller

Sam Sampson Lightstone

Liping Zhang

Richard Sefton Sidle

Yinan Li

Vincent Kulandaisamy

Naresh K Chainani

Gregory R Stager

Wayne Jonleon Young

Adam J Storm

Bishwaranjan Bhattacharjee

Chandrasekaran Mohan

Mohammad Sadoghi Hamedani

Robert S Germain

Vincent Kulandai Samy

Embodiments of the invention relate to processing data records, and for a multi-phase partitioned data reduction. The first phase relates to processing data records and partitioning the records into a first partition of records having a common characteristic and a second partition of records that are not members of the first partition. The data records in each partition are subject to intra-partition data reduction responsive to a resource constraint. The data records in each partition are also subject to an inter-partition data reduction, also referred to as an aggregation to reduce a footprint for storing the records. Partitions and/or individual records are logically aggregated and a data reduction operation for the logical aggregation of records takes place in response to available resources.

Accurate partition sizing for memory efficient reduction operations

A method creating a compact hash table includes a thread executing using a processor for building a linear probe hash table based on allocating space for an array of keys. Payloads are inserted into the linear probe hash table. Keys from the array of keys are inserted into the linear probe hash table with linear probing. A linear probing bitmap is generated based on occupied locations in the linear probe hash table. A compact hash table is created based on the linear probe hash table and the linear probing bitmap.

Parallel build of non-partitioned join hash tables and non-enforced N:1 join hash tables

A method of orchestrated shuffling of data in a non-uniform memory access device including a plurality of processing nodes that connected by interconnects. The method includes running an application on a plurality of threads executing on the plurality of processing nodes. Data to be shuffled is identified from source threads running on source processing nodes among the processing nodes to target threads executing on target processing nodes among the processing nodes. The method further includes generating a plan for orchestrating the shuffling of the data among the all of the memory devices associated with the threads and for simultaneously transmitting data over different interconnects to a plurality of different target processing nodes from a plurality of different source processing nodes. The data is shuffled among all of the memory devices based on the plan and each processing node is capable of accessing data from first and second local memory devices.

Data shuffling in a non-uniform memory access device

A method includes logically organizing, by an object hierarchy processor, data objects in a first hierarchy. A portion of the data objects in the first hierarchy logically includes groupings of other data objects. The object hierarchy processor physically organizes the data objects across two or more types of memory in a second hierarchy. Another portion of the data objects in the second hierarchy physically includes groupings of other data objects. Groupings of the data objects in the second hierarchy are dynamically moved across the two or more types of memory. Levels of access of the data objects are tracked using a data structure that maps groupings of the data objects in the first hierarchy onto metadata information including combined access frequencies of the data objects, and current number of accessors to the data objects, in each grouping of the data objects.

Efficient performance of insert and point query operations in a column store

Embodiments include a method, system, and computer program product for encoding data while it is being processed as part of a query is provided. The method includes receiving a query request and determining a set of values associated with data to be encoded for completing the query request. The method also includes encoding those values such that any subsequent processing operations can be performed on the encoded values to complete the requested query. After performing the subsequent processing operations to complete the requested query, each value is decoded back to its original value.

On-the-fly encoding method for efficient grouping and aggregation

A method of orchestrated shuffling of data in a non-uniform memory access device that includes a plurality of processing nodes that are connected by interconnects. The method includes running an application on a plurality of threads executing on the plurality of processing nodes. Data to be shuffled is identified from source threads running on source processing nodes among the processing nodes to target threads executing on target processing nodes among the processing nodes. The method further includes generating a plan for orchestrating the shuffling of the data among the all of the memory devices associated with the threads and for simultaneously transmitting data over different interconnects to a plurality of different target processing nodes from a plurality of different source processing nodes. The data is shuffled among all of the memory devices based on the plan. The data includes operand data and operational state data of the source threads.

A method of orchestrated shuffling of data in a non-uniform memory access device that includes a plurality of processing nodes connected by interconnects. The method includes running an application on a plurality of threads executing on the plurality of processing nodes. Data to be shuffled is identified from source threads running on source processing nodes among the processing nodes to target threads executing on target processing nodes among the processing nodes. The method further includes generating a plan for orchestrating the shuffling of the data among the all of the memory devices associated with the threads and for simultaneously transmitting data over different interconnects to a plurality of different target processing nodes from a plurality of different source processing nodes. The data is shuffled among all of the memory devices based on the plan. Shifting the data-shifting table includes rotating a first ring with respect to a second ring.

A method for creating a compact hash table includes a thread executing using a processor for scanning a subset of keys and values, and for each key, performing a hash operation and inserting hashed keys into a bitmap structure. Cumulative population counts of keys and values are determined within the bitmap. The method repeats scanning the subset of the keys and values. The keys and values are inserted into a compacted array using the cumulative population counts.

A method for building a hash table over a subset of data in a data set includes partitioning a subset of keys and values in the data set into multiple partitions. A hash table is formed with space reserved for each partition based on cumulative counts for a number of keys and values in each partition of the multiple partitions. Each thread selects one or more partitions and inserts keys and values belonging to the selected one or more partitions into the hash table in the reserved space for those partitions. A compact hash table is created that includes a bitmap and a compacted army.

Parallel build of non-partitioned join hash tables and non-enforced n:1 join hash tables

A method includes joining data between at least two data sets. Values of one or more join attributes of each of the data sets is represented in a compressed form, indicated by an encoding scheme. A compression scheme for the one or more join attributes is dynamically selected.

Efficient join on dynamically compressed inner for improved fit into cache hierarchy

Embodiments relate to data shuffling by logically rotating processing nodes. The nodes are logically arranged in a two or three dimensional matrix. Every time two of the nodes in adjacent rows of the matrix are aligned, adjacent nodes exchange data. The positional alignment is a logical alignment of the nodes. The nodes are logically arranged and rotated, and data is exchanged in response to the logical rotation.

Logical data shuffling

Embodiments relate to payload storage format for storing data in support of an aggregation function. As an input is subject to aggregation, the input is evaluated to ascertain a payload format for the aggregation. It is understood that there is more than one payload format. An evaluation of the aggregation key is a factor in the initial payload format. If the key is an addition to an existing aggregation, the evaluation considers changing the format of the payload to address processing and/or memory efficiency for the aggregation. The evaluation and the format change takes place dynamically so that the aggregation may continue.

Adaptive payload management

A method of orchestrated shuffling of data in a non-uniform memory access device that includes a plurality of processing nodes that are connected by interconnects. The method includes running an application on a plurality of threads executing on the plurality of processing nodes. Data to be shuffled is identified from source threads running on source processing nodes among the processing nodes to target threads executing on target processing nodes among the processing nodes. The method further includes generating a plan for orchestrating the shuffling of the data among the all of the memory devices associated with the threads and for simultaneously transmitting data over different interconnects to a plurality of different target processing nodes from a plurality of different source processing nodes. The data is shuffled among all of the memory devices based on the plan.

A method for building a hash table over a subset of data in a data set includes mapping keys in the data set to values in the data set using multiple parallel computation threads. Each thread scans a subset of the keys and values and partitioning the subset of the keys and values into multiple partitions. A cumulative count for keys and values in each partition is determined. A hash table with space reserved for each partition is formed based on the determined cumulative counts. Each thread selects one or more partitions and inserts keys and values belonging to the selected one or more partitions into the hash table in the reserved space for those partitions.

A method includes scanning multiple incoming database transaction requests. Each transaction includes one or more operations. Operations are clustered into a set of combined operations based on type of operation constraints. Log records are prepared and written for re-performing operations upon system failures, and for undoing operations upon an operation or a transaction failing to be processed fully. Each set of combined operations are performed within a thread. Each update operation is marked for a transaction within which the update operation belongs. Recoverable update operations belonging to a plurality of transactions are performed within a single logical thread of execution.

Growing community of inventors

Palo Alto, CA, United States of America

Ippokratis Pandis