Dell Products Inc.

Amit Zaitman

Uri Shabi

Alexander Shknevsky

Geng Han

Aleksey Kabishcher

Techniques for keeping on-disk, physical deduplication ('dedupe') index levels up to date to increase the efficiency of data deduplication in storage systems. The techniques include providing a plurality of dedupe indexes across a memory level and a disk level of a storage system, in which the plurality of dedupe indexes include an in-memory dedupe index at the memory level, and an on-disk dedupe index at the disk level. The techniques include applying a plurality of eviction rules regarding evicting entries of the on-disk dedupe index, in which the plurality of eviction rules avoid random evictions of entries from the on-disk dedupe index. The techniques include, in response to applying the plurality of eviction rules, identifying a plurality of entries of the on-disk dedupe index as eviction candidates. The techniques include evicting the identified eviction candidates from the on-disk dedupe index.

Method to keep on-disk physical deduplication index layer up to date

A method of data deduplication during log flushing includes use of an in-memory deduplication index as a cached sub-set of entries from a full on-drive deduplication index. In a pre-flush process, the data is scanned and the on-drive index is accessed to identify matching deduplication entries, and identified entries are copied into the in-memory index. The subsequent flush includes, for each data unit, (1) accessing the in-memory index to identify a matching deduplication entry for the data unit, (2) upon identifying a matching entry, storing the data unit in a deduplicated manner as a reference to previously stored data, and (3) upon not identifying a matching entry, storing the data unit in a non-deduplicated manner including a new data instance, without accessing the on-drive index. Performance and efficiency are improved by accessing only the in-memory index and avoiding the penalties of accessing the on-drive index.

Data storage system with streamlined deduplication during write log flushing

Techniques for achieving efficient removal of stale index entries from on-drive deduplication indexes using hash prefix indexing. The techniques include providing an on-drive dedupe index that includes a plurality of index entries. Each index entry includes a hash value of a data page, and an address associated with a storage location of the data page. Each index entry is assigned to a bucket defined by a respective hash prefix. The techniques include, for each data page associated with a reference count decremented to zero, logging a hash prefix and a storage address of the data page. The techniques include, for each bucket, constructing an address bag, storing, in the address bag, logged addresses whose corresponding hash prefix is the same as the respective hash prefix defining the bucket, and removing, from the bucket, each stale index entry that includes an address matching one of the addresses in the address bag.

Efficient removal of stale entries from on-drive deduplication hash tables using hash prefix indexing

A flush set of data pages to flush from a cache to non-volatile data storage is identified, and a compression group is identified that is made up of similar data pages within the flush set. A previously compressed data page is identified in non-volatile data storage that is similar to the data pages in the compression group. Data compression is performed on the compression group using a compression dictionary for the previously compressed data page. A flush set specific hash index and a global hash index may be used to identify the similar data pages in the compression group and the similar previously compressed data page. An indication of the similar previously compressed data page may be stored in the compressed block generated for the compression group, and the compression dictionary for the previously compressed data page need not be duplicated in the non-volatile data storage.

Growing community of inventors

Shavey Shomron, Israel

Amit Zaitman