Forming data shards to assign to deduplication domains in a deduplication file system

Abstract

A file is segmented into upper-level segments having references to lowest-level segments. Upper-level segments are aligned at fixed-size boundaries. First shards are formed. Each shard is aligned independent of an alignment of an upper-level segment and includes at least a subset of references from the upper-level segment. First shards are assigned to similarity groups within which deduplication is performed. Incremental changes are received. A virtual synthetic full is generated by segmenting new data into new upper-level segments. New upper-level segments are aligned at fixed-size boundaries and include new references to new segments of the new data. Second shards are formed. The shards include shards that include new references from the new upper-level segments and have new assignments to the similarity groups and other shards. The other shards include copies of at least subsets of references from the first shards and retain assignments to the similarity groups of the first shards.