Hierarchical wide spreading of distributed storage

Abstract

Systems and techniques for managing data storage are disclosed. In some aspects, a front-end node responds to a request to write an object by dividing the object into multiple source data segments. The front-end node generates redundancy data for the multiple source data segments using a rateless erasure encoding. The front-end node associates a respective subset of the redundancy data with each of the multiple source data segments, wherein each subset of redundancy data and associated source data segment form an encoded segment. The rateless erasure encoding further includes defining multiple segment-level fragments within each of the encoded segments. The front-end node transmits each of the encoded segments to a selected one of multiple storage nodes, wherein each of the selected storage nodes are selected based on a determined storage layout of the encoded segments across the multiple storage nodes. For each of the received encoded segments, the storage node generates one or more protection fragments based on redundancy data generated from the segment-level fragments and stores the segment-level fragments and corresponding protection fragments across multiple storage media devices managed by the selected storage node.