Netezza Corporation

Ibm International Group Bv

Netezza Corporaton

Huawei Technologies Co., Limited

Foster D Hinshaw

Barry M Zane

John K Metzger

David L Meyers

Craig S Harris

Premanand Yerabothu

Sunil Sarin

William K Stewart

Sharon Miller

James P Ballard

David A Utter

Steven T Almy

Raymond A Andraka

Venkannababu Tammisetti

Michael Sporer

Sanjay G Dixit

Philip J MacDonald

Dana A Kirkpatrick

Aaron C Beals

Vincent F Femia

Raymond J Andraka

A database appliance in which two or more internetworked data storage units are used to coordinate the storage and retrieval of database records. One or more central database processing units are also associated with the data storage units. A network infrastructure provides the ability for the central database processors and storage processors to communicate as network nodes, with the network infrastructure using a communication protocol. A software application for executing database operations executes in a distributed fashion with portions of the database application executing on at least one central database processor and other portions executing on the data storage processors. At least a portion of the database application is implemented within and/or coordinated by a communication process that is executing the communication protocol. This coordination takes place such that data blocks are passed between the communication process and at least one portion of the database application process by passing data block reference information. In accordance with other aspects of the present invention, the communication process may have at least portions of the database application process executing within it. These database application operations executing within the same context as the communication process may include database operations such as join, sort, aggregate, restrict, reject, expression evaluation, statistical analysis or other operations.

Network interface for distributed intelligence database system

A distributed data processing system executes database operations, and includes a first group of host processors and a second group of job processing units (JPUs). A software application for executing database operations executes in a distributed fashion with portions of the database application executing on at least one central database processor and other portions executing on the data storage processors. At least a portion of the database application is implemented within and/or coordinated by a communication process that is executing the communication protocol. The communication process implements a load balancing function in front of the host processors, the load balancing function directing individual transactions to selected processors so as to evenly distribute workload.

Field oriented pipeline architecture for a programmable data streaming processor

This invention relates generally to a system for processing database queries, and more particularly to a method for generating high level language or machine code to implement query execution plans. In one preferred embodiment, the method begins by receiving a subject query, and then forming an execution plan corresponding to the subject query. The execution plan will typically have a sequence of component snippets or pieces and corresponding processes for implementing the pieces. For at least one piece in the plan, the process then (a) generates source code using different code generation techniques as a function of expected runtime processing machine types; and then (b) compiles the generated source code to form machine executable code for implementing the subject query. As a result, the query executed directly as machine executable code thereby avoiding runtime interpretation of the pieces in the execution plan.

SQL code generation for heterogeneous environment

A system and method from processing database queries allows for cost and locale based distribution for execution of database queries. The database queries are executed on execution engines that provide flexible configuration and overlapping functionality. The system reduces various costs, including elapsed time, required to perform database queries. The method provides processing of a database query using a database catalog comprising database table locality information, record locality information and execution engine information. A query optimizer receives the query and accesses the catalog to create a query execution plan comprising locality-based database operations. A central database operation processor providing a first execution engine executes the query execution plan by performing at least a portion of the locality-based database operations and distributing at least a portion of the locality-based database operations as a subplan. A second database operation processor providing a second execution engine executes the subplan received from the central processor.

Optimized database appliance

A programmable streaming data processor that can be programmed to recognize record and field structures of data received from a streaming data source such as a mass storage device. Being programmed with, for example, field information, the unit can locate record and field boundaries and employ logical arithmetic methods to compare fields with one another or with values otherwise supplied by general purpose processors to precisely determine which records are worth transferring to memory of the more general purpose distributed processors. The remaining records arrive and are discarded by the streaming data processor or are tagged with status bits to indicate to the more general purpose processor that they are to be ignored. In a preferred embodiment, the streaming data processor may analyze and discard records for several reasons. The first reason may be an analysis of contents of the field. Other reasons for record blocking may have to do with tagging records that are to be visible to particular users depending upon a series of concurrent transactions.

Intelligent storage device controller

A field oriented pipeline processor for a data engine that can be programmed to recognize record and field structures of data received from a source such as a mass storage device. The processor has an interface for receiving field-delineated data from a field parser. The field parser parses non-field delineated data from a streaming data source into field delineated data under instruction from an external processing unit. The pipeline processor receives a field delineated data stream and employs logical arithmetic methods to compare fields with one another, or with values otherwise supplied by general purpose processors to precisely determine which records are worth transferring to memory of the more general purpose distributed processors. The architecture of the data engine allows for the use of substitution tables, temporary registers, and a data string register to assist in the efficiency and accuracy of the data engine processing.

In a preferred embodiment, the present invention is an asymmetric data processing system having two or more groups of processors that have attributes that are optimized for their assigned functions. A first processor group, which may be SMP machines, are responsible for interfacing with applications and/or end users to obtain queries, and for planning query execution. A second processor group consists of many streaming record-oriented processors called Job Processing Units (JPUs), typically arranged as an MPP structure. The JPUs carry out the bulk of the data processing required to implement the logic of a query, running autonomously and asynchronously from other processors in the system. The JPUs preferably use a multi-tasking operating system that permits multiple tasks to run at a given instant in time, in either an absolute-priority-based or a weighted-priority-based demand scheduling environment.

Asymmetric data streaming architecture having autonomous and asynchronous job processing unit

A data processing system having two or more groups of data processors that have attributes that are optimized for their assigned functions. A first group consists of one or more host computers responsible for interfacing with applications and/or end users to obtain queries and for planning query execution. A second processor group consists of many streaming record-oriented processors called Job Processing Units (JPUs), preferably arranged as an MPP structure. The JPUs typically carry out the bulk of the data processing required to implement the logic of a query. Each of the JPUs typically include a general purpose microcomputer, local memory, one or more mass storage devices, and one or more network connections. Each JPU also has a special purpose programmable processor, referred to herein as a Programmable Streaming Data Processor (PSDP). The PSDP serves as an interface between the CPU of a JPU and the mass storage device, to offload functions from the CPU of the JPU.

Programmable streaming data processor for database appliance having multiple processing unit groups

An asymmetric data record processor and method includes host computers and Job processing units (JPU's) coupled together on a network. Each host computer and JPU forms a node on the network. A plurality of software operators allow each node to process streams of records. For each operator in a given sequence within nodes and across nodes, output of the operator is input to a respective succeeding operator. Data processing follows a logical data flow based on readiness of a record. As soon as a record is ready it is passed for processing from one part to a next part in the logical data flow. The flow of records during data processing is substantially continuous and of a streaming fashion.

Asymmetric streaming record data processor method and apparatus

A system from processing database queries allows for cost and locale based distribution for execution of database queries. The database queries are executed on execution engines that provide flexible configuration and overlapping functionality. The system reduces various costs, including elapsed time, required to perform database queries. The system provides processing of a database query using a database catalog comprising database table locality information, record locality information and execution engine information. A query optimizer receives the query and accesses the database catalog to create a query execution plan comprising locality-based database operations. A central database operation processor providing a first execution engine executes the query execution plan by performing at least a portion of the locality-based database operations and distributing at least a portion of the locality-based database operations as a subplan. A second database operation processor providing a second execution engine executes the subplan received from the central database operation processor. At least one of the database operations can be executed on either the first execution engine or the second execution engine. A storage unit stores at least a portion of database tables and records. A data communications network connects the central database processor to the second database processor.

This invention relates generally to a system for processing database queries, and more particularly to a method for generating high level language or machine code to implement query execution plans. in one preferred embodiment, the method begins by receiving a subject query, and then forming an execution plan corresponding to the subject query. The execution plan will typically have a sequence of component snippets or pieces and corresponding processes for implementing the pieces. For at least one piece in the plan, the process then (a) generates source code using different code generation techniques as a function of expected runtime processing machine types; and then (b) compiles the generated source code to form machine executable code for implementing the subject query. As a result, the query executed directly as machine executable code thereby avoiding runtime interpretation of the pieces in the execution plan.

This invention relates generally to a system for processing database queries, and more particularly to a method for generating high level language or machine code to implement query execution plans. The present invention provides a method for generating executable machine code for query execution plans, that is adaptive to dynamic runtime conditions, that is compiled just in time for execution and most importantly, that avoids the bounds checking, pointer indirection, materialization and other similar kinds of overhead that are typical in interpretive runtime execution engines.

Optimized SQL code generation

A disk is segmented into a first data segment and a secondary data segment. The secondary data segment stores a logical mirror of the first data segment of another disk. Upon detecting a failure, the logical mirror of data stored in the first data segment of the failed disk is accessible from the secondary data segment of a non-failed disk. The first data segment can be rebuilt quickly on another disk from the logical mirror stored in the secondary data segment. During regenerating, accesses to the first data segment on the disk containing the logical mirror are handled by its own logical mirror, which is not involved in the regenerating process.

Disk mirror architecture for database appliance with locally balanced regeneration

A multi-version database system controls visibility of data during transaction processing. A transaction includes a transaction identifier that identifies the transaction and an invisibility list of transactions whose effects are invisible to the transaction. Changes made by other transactions are visible to the transaction based on the isolation level of the transaction. Records are visible to a transaction based on a creator identifier stored in the record that identifies the transaction that created the record.

Controlling visibility in multi-version database systems

A database appliance having multiple internetworked data storage units and central database processing units. The central database processors and storage units communicate as network nodes. Execution of a distributed database application is coordinated by a communication process such that data blocks are passed by using data block reference information. Operations executing within the same context as the communication process may include join, sort, aggregate, restrict, reject, expression evaluation, statistical analysis or other database operations.

Growing community of inventors

Somerville, MA, United States of America

Foster D Hinshaw