Layered distributed storage system and techniques for edge computing systems

Abstract

A two-layer erasure-coded fault-tolerant distributed storage system offering atomic access for read and write operations is described. In some embodiments, a class of erasure codes known as regenerating codes (e.g. minimum bandwidth regenerating codes) for storage of data in a backend layer is used to reduce a cost of backend bulk storage and helps in reducing communication cost of read operations, when a value needs to be recreated from persistent storage in the backend layer. By separating the functionality of edge layer servers and backend servers, a modular implementation for atomicity using storage-efficient erasure-codes is provided. Such a two-layer modular architecture permits protocols needed for consistency implementation to be substantially limited to the interaction between clients and an edge layer, while protocols needed to implement erasure code are substantially limited to interaction between edge and backend layers.