Fast incremental column store data loading

Abstract

A database is partitioned into a plurality of sorted runs, wherein each sorted run includes a set of sorted segments of data records. One of the sorted runs preferably includes more than half of the records of the database, and the other sorted runs are progressively smaller. A query is processed by searching each of the sorted runs. Writes are effected by appending a new sorted run to the database. Sorted merges are used to combine the smaller sorted runs. Deletions are effected by marking the deleted record in the sorted run. Modifications are effected by deleting the original record and writing the modified record to the database. The larger sorted runs are only re-sorted or merged when the sorted run includes a substantial number of deletions. Two merge queues are maintained to enable rapid merges of the smaller sorted runs while a merger of larger sorted runs are occurring.