Western Digital Technologies, Inc.

Amplidata N.v.

Koen De Keyser

Bastiaan Stougie

Carl Rene D'Halluin

Frederik De Schrijver

Frederik Jacqueline Luc De Schrijver

Stijn Blyweert

Thomas Demoor

Mark Christiaens

Cyril Guyot

Robert Eugeniu Mateescu

Adam Manzanares

Wim De Wispelaere

Lluis Pamies-Juarez

A novel distributed data storage system is disclosed. In an example method, a first plurality of key entries is stored in a first key data store at a first location and a second plurality of key entries is stored in a second key data at a second location. A key entry in comprises a corresponding key having an object identifier, an inverse timestamp, and a source identifier. The method further replicates a set of the first key entries to the second key data store. The method further inserts each first key entry from the set of the first key entries into the second key data store based on the object identifier, the inverse timestamp, and the source identifier of the first key included in that first key entry, the first key entries and the second key entries being interwoven to form a plurality of interwoven ordered key entries.

Conflict resolution and garbage collection in distributed databases

In an example embodiment, a method comprises determining that a multipart upload request to upload a data object in separate object parts has been received by an object storage service; generating temporary keys for the separate object parts of the data object; storing the temporary keys in a temporary key data store; generating, based on the temporary keys, a multipart key entry for the data object, the multipart key entry comprising a multipart key that contains an object identifier identifying the data object and an inverse timestamp; and inserting the multipart key entry in a persistent key data store storing an ordered set of key entries in a position determined by the object identifier and the inverse timestamp.

Key data store garbage collection and multipart object management

In an example embodiment, a method comprises determining an ordered set of key entries; determining a first key entry for a first object in the ordered set of key entries; determining an object storage operation represented by a key of the first key entry; determining the object storage operation represented by the key of the first key entry to comprise a delete operation; and responsive to determining the object storage operation represented by the key of the first key entry to comprise the delete operation, skipping over subsequent key entries associated with the first object in the ordered set of key entries.

Efficient key data store entry traversal and result generation

Novel key ticketing technology includes an example method in which a first request associated with a first object storage operation is received. The first request includes a first timestamp associated with the first object storage operation and a first object identifier identifying a first object associated with the first object storage operation. The method calculates a first inverse timestamp based on the first timestamp, and generates a first object key corresponding to the first object storage operation. The first object key includes at least the first object identifier and the first inverse timestamp. The method further inserts a first entry including the first object key into a key data store at a position relative to other object key entries based on the first object identifier and the first inverse timestamp included in the first object key.

Key ticketing system with lock-free concurrency and versioning

An object is divided into SDfirst-level pieces. Each first-level piece is stored in a first-level container on a first-level storage entity. A redundant encoding of the first-level containers is stored in RLadditional first-level containers on RLadditional first-level storage entities. On each of the first-level storage entities, the locally-stored first-level container is divided into SDlocal second-level pieces. Each second-level piece is stored in a second-level container on a second-level storage entity of the specific first-level storage entity. Each first-level storage entities contains SDplus RLsecond-level storage entities. A redundant encoding of the second-level containers is stored in RLadditional second-level containers on RLadditional second-level storage entities. The stored object is retrievable where any specific RLones of the first level storage entities are not available, and any specific RLones of the second-level storage entities of each remaining first-level storage entity are not available.

Multi-layer encoding for distributed container-based storage of smaller data objects

A data object can be encoded into a plurality of encoded data fragments and stored on backend storage elements in a distributed encoded storage system. The identifiers and metadata corresponding to each encoded fragment of the data object can be stored in a single metadata unit, which is stored on the backend as encoded fragments. The identifiers of the metadata fragments can be associated with the data object and stored on a low latency frontend storage device. Thus, the amount of metadata per data object stored on expensive low latency frontend storage is reduced to the fragment identifiers. The fragment identifiers can be quickly retrieved, and used to retrieve the identifiers and metadata corresponding to the encoded data fragments from the backend, for retrieval of the data object itself.

Reduced storage of metadata in a distributed encoded storage system

An ordered data object identifier denoted by a value is assigned to each data object grouped to a container object, wherein data object identifiers with successively incremented values are assigned to successive data objects. An ordered container identifier is assigned to each container object with the greatest value of the identifiers of the data objects grouped thereto. A metadata structure with an entry for each container objects is stored. Each entry comprises the ordered container identifier and a reference to the corresponding data. The metadata structure is ordered according to the values of the container identifiers. A request to read a data object contains a corresponding data object identifier. It is determined to which container object the data object is grouped, by locating the first container identifier in the metadata structure with a value greater than or equal to that the requested data object.

Utilization of optimized ordered metadata structure for container-based large-scale distributed storage

A data object can be encoded into multiple encoded data fragments and stored on the backend of a distributed encoded storage system. The identifiers and metadata corresponding to each fragment of the data object can be stored in a first section of a metadata unit, and the initial part of the data object in a second section. The metadata unit is encoded into multiple metadata fragments, which are stored on the backend. The identifiers of the metadata fragments can be associated with the data object and stored on a fast frontend storage device. In response to a request to access the data object, the identifiers are used to retrieve the metadata fragments from the backend, and decode the metadata unit. The initial part of the data object is retrieved from the metadata unit and transmitted to the requesting client application to begin processing the data object.

Fast read operation utilizing reduced storage of metadata in a distributed encoded storage system

A distributed object storage system has a monitoring agent and/or a maintenance agent configured to determine for each of a plurality of repair tasks the actual concurrent failure tolerance of a corresponding repair data object. The actual concurrent failure tolerance corresponds to the number of storage elements that store sub blocks of the repair data object and are allowed to fail concurrently.

Distributed object storage system

Techniques for distributing data in a distributed data storage system using a hierarchy rule that is generated based on a spreading policy and a set of tolerable failures specified by a user in absence of system deployment information are disclosed. The system includes a controller node which receives a request including a spreading policy and a protection level for spreading a first data object. The controller node determines a hierarchy rule corresponding to the spreading policy based on the protection level. The controller node distributes the first data object in the system using the hierarchy rule and the spreading policy. The controller node receives a reconfiguration of system deployment. The controller node distributes a second data object in the system based on providing protection of the protection level to the second data object without affecting protection of the same protection level applied to the first data object.

Hierarchic storage policy for distributed object storage systems

There is provided a distributed object storage system that includes several performance optimizations with respect to efficiently storing data objects when coping with a desired concurrent failure tolerance of concurrent failures of storage elements which is greater than two and with respect to optimizing encoding/decoding overhead and the number of input and output operations at the level of the storage elements.

Hybrid distributed storage system

A set of encoded data fragments is grouped into a container object in sequential order. Each encoded data fragment is a specific fragment size, and the container object is a specific container object size. The sequential order of the set of encoded data fragments can be tracked in a log in memory, such that the location of any one of the data fragments in the container object can be determined. The container object can be stored directly on a specific backend storage element, without using a file system. A corresponding container object identifier identifies the physical storage location of the container object on the backend storage element. The container object identifier is tracked in the log in memory, such that the physical location on the backend storage element of any specific one of the set of encoded data fragments in the container object can be determined.

Staging of write operations for container-based storage for sequential media

Small objects are efficiently stored with erasure codes by combining a small object with other small objects and/or large objects to form a single large object for chunking, and providing early notification of permanent storage to the sources of the objects to prevent small objects from becoming stale while waiting for additional objects to be combined.

Growing community of inventors

Sint-Denijs-Westrem, Belgium

Koen De Keyser