International Business Machines Corporation

Globalfoundries Inc.

Sam Sampson Lightstone

Adam J Storm

Richard Sefton Sidle

Vijayshankar Raman

Guy Maring Lohman

Ronald Jason Barber

Christian Marcelo Garcia-Arellano

Daniele Costante Zilio

Liping Zhang

Ippokratis Pandis

Matthew Albert Huras

Joseph L Hellerstein

Lin Qiao

Naresh K Chainani

Vincent Kulandaisamy

Sriram Kolinjivadi Padmanabhan

Gopi K Attaluri

Maheswaran Surendra

Yixin Diao

Sujay Sunil Parekh

Shaorong Liu

Min-Soo Kim

Ivan Popivanov

Kevin R Rose

Matthew James Carroll

Li Zhou

James L Finnie

Gary Valentin

Christina Marie Lee

Antti-Pekka Liedes

Marko Milek

Gregory R Stager

Wayne Jonleon Young

Ying-Pong Chen

Vijayshanker Raman

Jussi Vuorento

Wojciech Kuczynski

Manish Anand Bhide

Calisto Paul Zuzarte

Bishwaranjan Bhattacharjee

Peter Haas

Eugene Jon Shekita

Thomas Abel Beavin

Timothy Ray Malkemus

David C Kalmuk

Sumit Negi

Volker Gerhard Markl

Jun Rao

Laurent Sebastien Mignet

Daniel Costante Zilio

William T O'Connell

Randall William Horman

Xing Yi Chen

Miroslaw Adam Flasza

Paul C Huffman

Ashraf Aboulnaga

Haider Abbas Rizvi

Roger Luo Quan Zheng

Mark Donald Leitch

Jae Gil Lee

Richard E Swagerman

Sarah Posner

Robin D Van Boeschoten

Vincent Kulandai Samy

David Louis Kaminsky

Mir Hamid Pirahesh

Bruce Gilbert Lindsay

Paul Miller Bird

Roberta Jo Cochrane

James Alan Ruddy

Xun Xue

James C Kleewein

Michael Jeffrey Winer

Walid Rjaibi

Tuong Chanh Truong

William H Tetzlaff

Elizabeth Belva Hamel

John A McPherson, Jr

Catherine Suzanne McArthur

David Tremaine

Leslie Anne Cranston

James Liam Finnie

Bernhard K Schiefer

Michael Joshua Snowbell

Leo Tat Man Lau

Carolyn Haibt Norton

Aaron B Brown

Petrus Kai Chung Chan

Robert T Berks

Fei Yen Chiang

Tsz-Kin Lau

Adriana Zubiri

Lan Pham

Peter K L Shum

Michael T Ho

Said Elnaffar

Pat Martin

Viyayshankar Raman

Alain C Comeau

John Ai Mcpherson

Embodiments of the invention relate to processing data records, and for a multi-phase partitioned data reduction. The first phase relates to processing data records and partitioning the records into a first partition of records having a common characteristic and a second partition of records that are not members of the first partition. The data records in each partition are subject to intra-partition data reduction responsive to a resource constraint. The data records in each partition are also subject to an inter-partition data reduction, also referred to as an aggregation to reduce a footprint for storing the records. Partitions and/or individual records are logically aggregated and a data reduction operation for the logical aggregation of records takes place in response to available resources.

Accurate partition sizing for memory efficient reduction operations

In updating a synopsis table of a database system, a database management unit performs a transaction to insert row(s) in a section of the base table and determines whether a synopsis entry for the section is stored in the memory. If stored in the memory, the in-memory synopsis entry is retrieved and metadata values in the in-memory synopsis entry are updated with data from the row(s) to be inserted. If not stored in the memory, the in-memory synopsis entry is generated and the metadata values in the in-memory synopsis entry are updated with data from the row(s). The insert transaction is then committed. Synopsis entry on-disk updates are thus avoided, significantly reducing the cost of updating the synopsis entries from the insert transaction. This yields enhanced performance especially for inserts of a small number of rows, while the benefits of synopsis entries are still available.

Updating of in-memory synopsis metadata for inserts in database table

Performing a join of first and second database tables for a query includes applying a predicate of the query to the first table as a first predicate and determining how many distinct join key values the first table has that survive the applying of the first predicate, wherein a join key value of the first table that survives the applying of the first predicate is a surviving join key value for a second predicate. A selection includes selecting among applying the second predicate to the second table, probing the second table with the second predicate, and neither applying the second predicate to the second table nor probing the second table with the second predicate, wherein the selecting is responsive to the number of distinct, surviving join key values.

Technology for join processing

Embodiments include a method, system, and computer program product for encoding data while it is being processed as part of a query is provided. The method includes receiving a query request and determining a set of values associated with data to be encoded for completing the query request. The method also includes encoding those values such that any subsequent processing operations can be performed on the encoded values to complete the requested query. After performing the subsequent processing operations to complete the requested query, each value is decoded back to its original value.

On-the-fly encoding method for efficient grouping and aggregation

A system joins predicate evaluated column bitmaps having varying lengths. The system includes a column unifier for querying column values with a predicate generating an indicator bit for each of the column values that is then joined with the respective column value. The system also includes a bitmap generator for creating a column-major linear bitmap from the column values and indicator bits. The column unifier also determines an offset between adjacent indicator bits. The system also includes a converter for multiplying the column-major linear bitmap with a multiplier to shift the indicator bits into consecutive positions in the linear bitmap.

Multiplication-based method for stitching results of predicate evaluation in column stores

In one embodiment, a method includes adding, by a computer processor, two or more compressed columns to one or more pages of a database. The adding is performed in parallel by a plurality of page-formatter threads. Each page-formatter thread adds data to the database from no more than a single compressed column.

Parallel load in a column-store database

The embodiments described herein relate to accessing a plurality of data elements. A page of column data is compressed and stored in a format that includes a collection of data elements. A tuple map is stored, and the collection of data elements is indexed via the tuple map. A query is processed based on the compressed page by identifying a set of tuple identifiers mapping to stored data in support of the query. Each tuple identifier corresponds to a location of a respective tuple of the compressed page.

Data encoding and processing columnar data

Embodiments relate to payload storage format for storing data in support of an aggregation function. As an input is subject to aggregation, the input is evaluated to ascertain a payload format for the aggregation. It is understood that there is more than one payload format. An evaluation of the aggregation key is a factor in the initial payload format. If the key is an addition to an existing aggregation, the evaluation considers changing the format of the payload to address processing and/or memory efficiency for the aggregation. The evaluation and the format change takes place dynamically so that the aggregation may continue.

Adaptive payload management

A computer-implemented method includes receiving a request to transmit column group data to a target node, the column group data comprising C columns within a column-oriented data table and determining a transmission row count R for transmitting the column group data to the target node. The method may also include transmitting a transmission packet comprising R sequentially-ordered data elements for each of the C columns to the target node. The R data elements for each column may be sequentially retrieved from memory. A corresponding method includes receiving, at a target node, a request to receive the column group data, determining the transmission row count R for receiving the column group data, and receiving a transmission packet comprising R sequentially-ordered data elements for each of the C columns and storing the sequentially-ordered data elements within memory. A corresponding computer system and corresponding computer program products are also disclosed herein.

Data transfers in columnar data systems

Embodiments of the invention relate to payload storage format for storing data in support of an aggregation function. As an input is subject to aggregation, the input is evaluated to ascertain a payload format for the aggregation. It is understood that there is more than one payload format. An evaluation of the aggregation key is a factor in the initial payload format. If the key is an addition to an existing aggregation, the evaluation considers changing the format of the payload to address processing and/or memory efficiency for the aggregation. The evaluation and the format change takes place dynamically so that the aggregation may continue.

A method for storing database information includes storing a table having data values in a column major order. The data values are stored in a list of blocks. The method also includes assigning a tuple sequence number (TSN) to each data value in each column of the table according to a sequence order in the table. The data values that correspond to each other across a plurality of columns of the table have equivalent TSNs. The method also includes assigning each data value to a partition based on a representation of the data value. The method also includes assigning a tuple map value to each data value. The tuple map value identifies the partition in which each data value is located.

Intra-block partitioning for database management

Growing community of inventors

Toronto, Canada

Sam Sampson Lightstone