Method and system for designing a database system for high event rate, while maintaining predictable query performance

Abstract

A computer-implemented database system, method, and computer program product are provided. the database system comprises a plurality of nodes, each node including a data storage device having memory and a server; and a database system controller coupled to each of the plurality of nodes and having a processor and software program code for directing the database system to perform the following function of defining, for a user, a node group including at least one of the plurality of nodes, upon which the user's data is to be stored in a user database; wherein the user database is defined in terms of time-partitioned tables residing on the nodes of the node group, and wherein each time-partitioned table is further defined in terms of shards of the user's data, the shards corresponding with respective ones of the time partitions. The user data is managed in terms of the shards on respective nodes of the node group, and the time-partitioned tables include a chronologically oldest time-partitioned table and a current time-partitioned table into which a new user data entry will be written. The system further performs the functions of determining whether a predetermined criterion has been met; and, responsive to the detecting that the predetermined criterion has been met, purging all shards of the chronologically oldest time-partitioned table.