Dell Products Inc.

Dell International L.l.c.

Ryan W Brosch

A data deduplication method includes detecting a deduplication transaction including a data pattern associated with a data pattern address (DPA) and a reference, to the pattern, associated with a data reference address (DRA). A deduplication key may be determined based on the DPA and the DRA by concatenating the DPA and the DRA with the DPA as the most significant bits. The key may be stored in a key field of a record in a persistent and sequentially-accessed log, which is part of a log-with-index (LWI) structure that also maintains, in RAM or SSD, a binary index of the log records. When full, the log is cleared by writing the records in key-sorted order to the new tablet. From time to time, two tablets in the tablet library are merged. Tablet merging may include two or more atomic merges, each atomic merge corresponding to a portion of the tablet.

Maintaining data deduplication reference information

An information handling system maintains a log-with-index (LWI) structure that includes a log for inserting and deleting records, each of which includes a key, and a binary index for maintaining an index of log records. The log may be stored to disk for persistence. Insertions of new records into the log may occur sequentially to ensure adequate performance. The index tree may be maintained in memory for performance. When the log is full, log records may be written in key-sorted order to a new tablet in a tablet library. Two tablets may be merged from time to time. Merging may include iteratively performing a plurality of atomic merges for each of a plurality of atomic portions. During atomic merges, tablet index data may be modified in copy-on-write fashion to preserve existing data until the atomic merge completes.

Maintaining I/O transaction metadata in log-with-index structure

A data deduplication process maintains a data dictionary including a storage tablet and a secondary index containing data indicative of previously received data blocks. The tablet includes hashes of previous data blocks and the index includes one or more cuckoo filters storing fingerprints derived from block hashes of previous data blocks. When a new data block arrives, its block hash and fingerprint are generated. The storage tablet is queried with the block hash and the secondary index is queried with the fingerprint. If the dictionary contains no matching block hash or fingerprint, the new data block is stored in its entirety. If the dictionary contains a matching block hash or fingerprint, the new data block may be a duplicate data block that can be deduplicated by storing a reference to the previous data block instead of storing the new data block in its entirety.

Data deduplication with augmented cuckoo filters

A method including maintaining a library having a plurality of storage tablets, each storage tablet storing a plurality of hash-to-storage mappings, each mapping a hash value to a storage location at which a block of data is stored, the block of data translating to the hash value pursuant to a hashing algorithm. The method also including upon receipt and/or determination of a new hash for incoming data pursuant to the hashing algorithm: a) querying a tablet cache for a hash-to-storage mapping having the new hash, the tablet cache comprising a subset of storage tablets copied from the library; and/or b) querying a secondary index for a hash-to-storage tablet mapping having the new hash, the secondary index including a plurality of filters, each filter mapping each of a plurality of key hashes to a storage tablet of the library storing that particular key hash in a hash-to-storage mapping.

Growing community of inventors

Arden Hills, MN, United States of America

Ryan W Brosch