Western Digital Technologies, Inc.

Quantum Corporation

Thomas Demoor

Carl Rene D'Halluin

Stijn Devriendt

Bastiaan Stougie

Koen De Keyser

Ewan Higgs

Koen Struyve

Turguy Goker

Don Doerner

Frederik De Schrijver

Arne De Coninck

Sam De Roeck

Wim Vander Schelden

Mary Hayes

Carl D'halluin

Viswanath Chandrasekara Bharathi

Satish Kumar Chakka

Stijn Blyweert

Example distributed storage systems, replication managers, and methods provide replication barriers for dependent data transfers between data stores. An object data store may include a barrier object and be configured to identify dependencies between a dependency set of data objects and the barrier object. When replicating data objects to another data store, the dependency set of data objects may be transferred first, delaying the transfer of the barrier object while the dependency set is being transferred.

Replication barriers for dependent data transfers between data stores

A method for utilizing an object storage system () for storing and retrieving a volume of data, the volume of data including a plurality of data objects, the method includes the steps of accumulating the volume of data to an ingest area () provided within a non-taped-based storage medium; copying a first subset of the plurality of data objects to a first tape medium (A); and copying a second subset of the plurality of data objects to a second tape medium (B) that is independent of the first tape medium (A). The method can further include managing the object storage system () with object storage software () to perform one or more of (i) generating a first extent parity that contains redundant information from the first subset of the plurality of data objects using erasure coding; and writing the first extent parity to the first tape medium (A); (ii) generating a second extent parity that contains redundant information from the second subset of the plurality of data objects using erasure coding; and writing the second extent parity to the second tape medium (B); and (iii) generating a volume parity that contains redundant information from the volume of data using erasure coding; and writing the volume parity to a third tape medium (C) that is independent of the first tape medium (A) and the second tape medium (B).

System and method for storing and retrieving objects across multiple media

Example storage systems and methods provide data storage management using parallel processing of filtered transaction logs. Transaction logs are comprised of log entries corresponding to storage operations for at least one storage node. Sets of log entries are sequentially retrieved from the transaction log and filtered through multiple transaction log filters to generate multiple subsets of the log entries. Different metadata operations are executed in parallel using the different filtered subsets of log entries.

Parallel processing of filtered transaction logs

Example distributed storage systems, delete managers, and methods provide for managing dependent delete operations among data stores. Dependent data operation entries and corresponding dependency sets may be identified in an operations log. Dependent data operations may be identified in each shard and data operation entries. A delete process for the data objects in the dependency set may be delayed until the delete process for the dependent data object completes.

Managing dependent delete operations among data stores

Example storage systems and methods provide data storage management using a key data store with progress values. A key data store includes a set of key data entries that each include a key value associated with a storage operation and a timestamp corresponding to a creation time of the key data entry. Storage management processes are executed on the set of key data entries and progress values for the storage management processes are tracked using the timestamps of the key data entries to manage the relative progress of the storage management processes.

Key value store using progress verification

Example distributed storage systems, replication managers, and methods provide barriers for dependent data operations among sharded data stores. Shard operation logs may be associated with different shards and storage nodes. Dependent data operations may be identified in each shard and data operation entries corresponding to dependency sets for those dependent data operations may be determined. Log-based operations may be executed by delaying execution of the dependent data operations in all shards until the dependency set data operations in all shards are completed.

Barriers for dependent operations among sharded data stores

Example storage systems and methods provide data storage management using change values in a key data store. A key data store includes a set of key data entries that each include a key value associated with a storage operation, a precondition value, and a postcondition value. A storage parameter is calculated using the precondition value and the postcondition value for each key data entry the set of key data entries. The calculated storage parameter may be reported to manage storage configuration and operations in the storage system.

Key value store using change values for data properties

Example storage systems, storage nodes, and methods provide storage node processing of data functions, such as map-reduce functions, using overlapping symbols. Storage nodes are configured to partition data units into symbols that include an overlap data portion of an adjacent symbol and erasure encode the symbols. The storage nodes may then decode erasure encoded symbols, identify subunits of a data unit from the decoded symbols, and process the subunits using map-functions to generate intermediate contexts. A reduce-function may be used to determine a function result using the intermediate contexts.

Storage node processing of data functions using overlapping symbols

Example storage systems, storage nodes, and methods provide storage node processing of data functions, such as serial functions. Storage nodes are configured to partition decode erasure encoded symbols, identify subunits of a data unit from the decoded symbols, process the subunits using a serial function to generate intermediate contexts, and send the intermediate context to a next storage node for continued processing using the serial function.

Serial storage node processing of data functions

Systems and methods for metadata compaction in a distributed storage system with a file system interface are described. A file system interface and an object storage system interface use a metadata index for mapping object identifiers from the object storage system to location identifiers for the file system. When the metadata index includes a number of entries for continuous data blocks with overlapping intervals, a defragmentation operation may generate a defragmented entry for a defragmentation interval overlapping the overlapping data blocks.

Metadata compaction in a distributed storage system

Example distributed storage systems, replication state engines, and methods manage replication state for guaranteed replication between data stores. An object data store may store data objects that have been determined for deletion and rendered inaccessible to a client application. A replication state may be queried for deleted data objects and, if the replication state indicates that replication to another object data store is incomplete, physical deletion may be delayed until the replication state indicates that replication is complete and the data object may be physically deleted.

Managing replication state for deleted objects

A novel distributed data storage system is disclosed. In an example method, a first plurality of key entries is stored in a first key data store at a first location and a second plurality of key entries is stored in a second key data at a second location. A key entry in comprises a corresponding key having an object identifier, an inverse timestamp, and a source identifier. The method further replicates a set of the first key entries to the second key data store. The method further inserts each first key entry from the set of the first key entries into the second key data store based on the object identifier, the inverse timestamp, and the source identifier of the first key included in that first key entry, the first key entries and the second key entries being interwoven to form a plurality of interwoven ordered key entries.

Conflict resolution and garbage collection in distributed databases

In an example embodiment, a method comprises determining that a multipart upload request to upload a data object in separate object parts has been received by an object storage service; generating temporary keys for the separate object parts of the data object; storing the temporary keys in a temporary key data store; generating, based on the temporary keys, a multipart key entry for the data object, the multipart key entry comprising a multipart key that contains an object identifier identifying the data object and an inverse timestamp; and inserting the multipart key entry in a persistent key data store storing an ordered set of key entries in a position determined by the object identifier and the inverse timestamp.

Key data store garbage collection and multipart object management

Example distributed storage systems, controller nodes, and methods provide selective allocation of redundant data blocks to background operations. Background operations may be identified targeting a data unit stored in redundant data blocks in a storage pool with a plurality of storage elements. A subset of data units may be selected for the background operation and the system components including those data units may be isolated. Data requests to the isolated system components may be selectively prevented while the background operation executes on the subset of data units in the isolated system components.

Selective allocation of redundant data blocks to background operations

Example distributed storage systems, file system interfaces, and methods provide duplicate request checking. A file interface manages a plurality of file data requests from client systems using a file system protocol. A metadata store acts as a cross-reference between file data references and storage data references. A request data store in the metadata store includes request entries with transaction identifiers for corresponding file data requests. A duplicate checker searches the request data store and excludes incoming file data requests when it finds a matching request entry.

Growing community of inventors

Ghent, Belgium

Thomas Demoor