Compaction of documents in a high density data storage system

Abstract

A system stores multiple documents in a log-structured object store, each document associated with a sequence number and a document value. The log-structured object store includes multiple log segments, each storing documents within a defined sequence number range. A first log-structured merge-tree is maintained to map document keys to corresponding sequence numbers, while a second log-structured merge-tree maps sizes of stale documents to their sequence numbers. A ratio is determined between the size of stale documents and the total size of documents in at least one log segment. In response to the ratio exceeding a predetermined threshold, a compaction operation is initiated to delete stale documents and consolidate active documents into a reduced number of log segments using the reclaimed space. The first and second log-structured merge-trees are updated based on the compaction results to reflect the current state of the data.