International Business Machines Corporation

Netezza Corporation

Ibm International Group Bv

David L Meyers

Foster D Hinshaw

Barry M Zane

Garth Allen Dickie

Eileen Tien Lin

William B Ackerman

William K Stewart

Daniel J Feldman

Sharon Miller

Shivin S Misra

Michael R Zabka

John K Metzger

Raymond A Andraka

Sanjay G Dixit

Venkannababu Tammisetti

Michael Sporer

Craig S Harris

Premanand Yerabothu

Vincent F Femia

Raymond J Andraka

Provided are techniques for processing window partitioning and ordering for OLAP functions. A prior compare operation is performed by: receiving an input vector for each of one or more attributes of input data that represent one of a partition-by column and an order-by column in a database query; generating a per-attribute comparison vector for each input vector; and producing a single output vector using each per-attribute comparison vector, wherein each value of the single output vector is a Boolean attribute whose value for a given tuple is true if a current value and a most recent prior value of any of the one or more attributes are different.

Processing window partitioning and ordering for on-line analytical processing (OLAP) functions

Metadata for a plurality of database storage regions within memory are merged, where the metadata for each storage region comprises an interval including first and second interval values indicating a value range for values within that storage region. The first and second interval values are examined to identify overlapping storage regions and produce a sum of over-lapped storage regions. The sum of overlapped storage regions is compared to a threshold and the metadata of the overlapped storage regions are merged based on the comparison.

Merging metadata for database storage regions based on overlapping range values

Metadata for a plurality of database storage regions within memory are merged, where the metadata for each storage region comprises an interval including first and second interval values indicating a value range for values within that storage region. The first and second interval values are examined to identify overlapping storage regions and produce a sum of overlapped storage regions. The sum of overlapped storage regions is compared to a threshold and the metadata of the overlapped storage regions are merged based on the comparison.

A distributed data processing system executes database operations, and includes a first group of host processors and a second group of job processing units (JPUs). A software application for executing database operations executes in a distributed fashion with portions of the database application executing on at least one central database processor and other portions executing on the data storage processors. At least a portion of the database application is implemented within and/or coordinated by a communication process that is executing the communication protocol. The communication process implements a load balancing function in front of the host processors, the load balancing function directing individual transactions to selected processors so as to evenly distribute workload.

Field oriented pipeline architecture for a programmable data streaming processor

A programmable streaming data processor that can be programmed to recognize record and field structures of data received from a streaming data source such as a mass storage device. Being programmed with, for example, field information, the unit can locate record and field boundaries and employ logical arithmetic methods to compare fields with one another or with values otherwise supplied by general purpose processors to precisely determine which records are worth transferring to memory of the more general purpose distributed processors. The remaining records arrive and are discarded by the streaming data processor or are tagged with status bits to indicate to the more general purpose processor that they are to be ignored. In a preferred embodiment, the streaming data processor may analyze and discard records for several reasons. The first reason may be an analysis of contents of the field. Other reasons for record blocking may have to do with tagging records that are to be visible to particular users depending upon a series of concurrent transactions.

Intelligent storage device controller

A field oriented pipeline processor for a data engine that can be programmed to recognize record and field structures of data received from a source such as a mass storage device. The processor has an interface for receiving field-delineated data from a field parser. The field parser parses non-field delineated data from a streaming data source into field delineated data under instruction from an external processing unit. The pipeline processor receives a field delineated data stream and employs logical arithmetic methods to compare fields with one another, or with values otherwise supplied by general purpose processors to precisely determine which records are worth transferring to memory of the more general purpose distributed processors. The architecture of the data engine allows for the use of substitution tables, temporary registers, and a data string register to assist in the efficiency and accuracy of the data engine processing.

A data processing system having two or more groups of data processors that have attributes that are optimized for their assigned functions. A first group consists of one or more host computers responsible for interfacing with applications and/or end users to obtain queries and for planning query execution. A second processor group consists of many streaming record-oriented processors called Job Processing Units (JPUs), preferably arranged as an MPP structure. The JPUs typically carry out the bulk of the data processing required to implement the logic of a query. Each of the JPUs typically include a general purpose microcomputer, local memory, one or more mass storage devices, and one or more network connections. Each JPU also has a special purpose programmable processor, referred to herein as a Programmable Streaming Data Processor (PSDP). The PSDP serves as an interface between the CPU of a JPU and the mass storage device, to offload functions from the CPU of the JPU.

Programmable streaming data processor for database appliance having multiple processing unit groups

An asymmetric data record processor and method includes host computers and Job processing units (JPU's) coupled together on a network. Each host computer and JPU forms a node on the network. A plurality of software operators allow each node to process streams of records. For each operator in a given sequence within nodes and across nodes, output of the operator is input to a respective succeeding operator. Data processing follows a logical data flow based on readiness of a record. As soon as a record is ready it is passed for processing from one part to a next part in the logical data flow. The flow of records during data processing is substantially continuous and of a streaming fashion.

Asymmetric streaming record data processor method and apparatus

A disk is segmented into a first data segment and a secondary data segment. The secondary data segment stores a logical mirror of the first data segment of another disk. Fast access to data stored on the disk is provided by partitioning the disk such that the first data segment includes the fast tracks of the disk and the secondary data segment includes the slow tracks of the disk and forwarding all data requests to the first data segment. Upon detecting a failure, the logical mirror of data stored in the first data segment of the failed disk is accessible from the secondary data segment of a non-failed disk. The first data segment can be rebuilt quickly on another disk from the logical mirror stored in the secondary data segment.

Growing community of inventors

Shrewsbury, MA, United States of America

David L Meyers