Emc IP Holding Company LLC

Andrey Vsevolodovich Kurilov

Mikhail Danilov

Konstantin Buinov

Alexander Rakulenko

Kirill Viktorovich Gusakov

Andrey Fomin

Maxim S Trusov

Kirill Zakharov

Mikhail Malygin

Gregory Skripko

Ivan Tchoub

Irina Tavantseva

Aleksander Georgievich Rakulenko

Scaling-in of a geographically diverse data storage system is disclosed. Data chunks stored in storage devices of different zones can be convolved to conserve memory. The zone storage system can be a geographically diverse storage system. Removing a zone from the geographically diverse data storage system can risk data loss without adequate replication of data from the zone to be removed. The zone to be removed can be placed in a 'read-only' state. Data chunks form the zone to be removed can be duplicated at other zones of the geographically diverse data storage system. A data chunk comprising convolved representations of data is duplicated to a zone in a manner that results in loss of geographical diversity, a corresponding chunk can be duplicated at another zone to restore geographical diversity. Validation of data can be performed prior to removal of a zone in a scaling-in event.

Scaling-in for geographically diverse storage

A computer program product, system, and method for generating a test data set using object count and object size parameters, storing the generated data set in storage, subsequently regenerating the data set, and comparing the regenerated data set to the stored data set to validate the storage.

Data set verification

Methods and apparatus are provided for protection of state information for an auxiliary storage service in a microservice architecture. An exemplary method comprises generating a snapshot of state information of an auxiliary storage service on a given storage node in a storage cluster comprised of a plurality of storage nodes; and providing the snapshot to M protector storage nodes within the plurality of storage nodes in the storage cluster, wherein the M protector storage nodes comprises a snapshot manager node and M&#x2212;1 additional protector storage nodes, and wherein the M protector storage nodes are selected based on a hierarchical ranking of available storage nodes within the storage cluster arranged in a predefined configuration relative to the given storage node. The predefined configuration of the plurality of storage nodes can be, for example, a protection ring comprising a first M&#x2212;1 online storage nodes that follow the snapshot manager node in the protection ring in a predefined direction.

Management of state information backup for an auxiliary storage service in a microservice architecture

Method for tree management of trees under multi-version concurrency control is described herein. Method starts by detecting change in a size of a cluster. The number of nodes in the cluster is counted to obtain the changed size of the cluster. The number of trees needed for the number of nodes in the cluster is determined. The number of trees may be based on the number of nodes in the cluster and predetermined system balancing coefficient. When the number of trees needed is greater than existing number of trees, existing number of trees is doubled as a single-step operation which includes modifying and scaling hash function used to derive each tree to generate new hash function and using new hash function to partition each tree instantly into two new trees. Scaling happens on demand without service disruption. Hash function scales automatically when number of trees increases. Other embodiments are described.

Method and system for tree management of trees under multi-version concurrency control

Scale out data protection with erasure coding is presented herein. Based on an initial number of storage devices determined to have been included in an initial stage of a data storage cluster, an initial protection scheme for the initial stage can determine first coding fragment(s) for data stored within the data storage cluster to facilitate a first recovery, from the initial stage, of the data using the first coding fragment(s). Further, in response to a defined number of additional storage devices being determined to have been added to the data storage cluster to generate a modified data storage cluster, the initial protection scheme can be modified to obtain a modified protection scheme that can determine, for the modified data storage cluster, second coding fragment(s) for the data to facilitate a second recovery of the data using the first coding fragment(s) and the second coding fragment(s).

Scale out data protection with erasure coding

Overhead associated with data re-protection during scaling out and/or scaling up of a cloud storage system can be reduced. During initial data protection (e.g., prior to a change in the cluster size), data fragments and coding fragments can be stored as a set of adjacent data sub-fragments and a set of adjacent coding sub-fragments, respectively. In response to determining that the cluster has expanded, a portion of the set of adjacent data sub-fragments can be moved to the new nodes that have been added to the cluster. Further, the set of adjacent coding sub-fragments can be combined to generate a new coding fragment that can be stored within the new cluster. Accordingly, the amount of system resources utilized during re-protection can be significantly reduced.

Scale-out erasure coding

One embodiment is related to a method for implementing a demoted chunk, comprising: replicating a first chunk from a first zone of a cloud storage system to a third zone of the cloud storage system; replicating a second chunk from a second zone of the cloud storage system to the third zone; creating an exclusive or (XOR) chunk at the third zone based at least in part on the replicated first chunk and the replicated second chunk; and marking the replicated first chunk and the replicated second chunk as demoted first chunk and demoted second chunk, respectively, at the third zone.

System and method for capacity and network traffic efficient data protection on distributed storage system

A method for creating distributed erasure coding chunks in a distributed storage system with unbalanced load is disclosed. The operations comprise configuring the distributed storage system into at least k+m zones, wherein each zone accumulates at least l primary backup chunks of original data chunks replicated from different remote zones, and wherein l&#x3c;k, and preparing the distributed storage system for recovery from a failure of 1 to m zones of the at least k+m zones including, in each zone, encoding the at least l primary backup chunks to create m coding chunks using an erasure coding having parameters k+m, wherein in each zone in which fewer than k primary backup chunks have been accumulated, at most k&#x2212;l predetermined virtual chunks are used to create partial coding chunks.

System and method for unbalanced load handling with distributed erasure coding

A method comprising: generating a storage chunk having a plurality of data fragments, the storage chunk stored in one or more storage devices; allocating space in a primary memory to store a plurality of coded fragments; receiving a plurality of I/O requests to write data; allocating space in the primary memory to store a next unfilled data fragment; processing a plurality of I/O requests to write data; and copying the coded fragments from the primary memory to the one or more storage devices. For each I/O request, writing the data to the next unfilled data fragment in the one or more storage devices, writing the data to the next unfilled data fragment in the primary memory, and if the data fragment in the primary memory is full, updating the coded fragments in the primary memory using the filled data fragment in the primary memory.

Incremental erasure coding for storage systems

A method comprising: receiving an I/O request for object data; determining one or more data fragments wherein the object data is stored; determining that one or more of the data fragments are unavailable; determining, from within the one or more unavailable data fragments, a set of slices storing the object data, each slice comprising k small data fragments and m coded fragments; for each slice, retrieving at least k small data and coded fragments within the slice from storage; and recovering a segment of the object data using the retrieved small data and coded fragments. A related system and computer program product are also described.

Data object recovery for storage systems

Data object content verification systems and processes provide perfect reliability and low storage overhead. Object data is generated in a reproducible manner based upon object locally stored object metadata. The object data is stored to an object storage system. The stored object data is subsequently verified by retrieving the object metadata, reproducing the original object data, and comparing the stored and original object data. The data object content verification systems and processes support both mutable and immutable data object content verification.

Mutable data objects content verification tool

A method for generating coded fragments comprises receiving data to be encoded, splitting the data into a plurality of data fragments, identifying a first group of data fragments from among the plurality of data fragments using a coding matrix, summing the data fragments within the first group of data fragments to generate a first group sum, and using the first group sum to calculate at least a portion of two or more coded fragments.

Accelerated erasure coding for storage systems

The disclosure relates to data protection management for geographically distributed storage systems. Specifically, the present disclosure provides a mechanism for load balancing by combining the techniques of XOR Star and XOR Chain to control the protection of data internally. The load balancing is done by each zone without the need for an external load balancer. In addition, the load balancing mechanism provides the ability for native (e.g. without additional configuration) support of disaster recovery.

Growing community of inventors

Vsevolozhsk, Russia

Andrey Vsevolodovich Kurilov