Detecting missing segments and identifying files impacted by the missing segments using a single pass distributed scan

Abstract

A method for managing data in a storage system includes: partitioning segments to generate buckets; assigning a portion of the buckets to a first storage node; collecting a partition of the segments assigned as child segments; identifying: a first set of child segments that maps to the portion of the buckets, and a second set of child segments that maps to a bucket of a second storage node; storing: each of the first set of child segments to an LSM tree of the first storage node, the second set of child segments to a buffer; sending the second set of child segments to the second storage node; receiving a third set of child segments from the second storage node; comparing the first set of child segments and third set of child segments of the LSM tree against second child segments; and making a determination that a mismatch is occurred.