Emc IP Holding Company LLC

Dell Products Inc.

Dell & Products L.p.

Other

Uri Shabi

Vladimir Shveidel

Ronen Gazit

Alex Soukhman

Maor Rahamim

Bar Harel

Vamsi K Vankamamidi

Geng Han

Anton Kucherov

Amitai Alkalay

Alexander Shknevsky

Alexei Kabishcer

Zvi Schneider

Amit Zaitman

Bar David

Dror Zalstein

Alon Titelman

Nir Bar Joseph

Xiangping Chen

Lior Kamran

Shaul Dar

Steven A Morley

Tal Ben-Moshe

Nimrod Shani

Samuel L Mullis, Ii

Michael Litvak

Mark Abashkin

Liang Huang

Jonathan Volij

Boaz Sela

Aleksey Kabishcher

Aleksey Kabishcer

Rachel Fraenkel-Saban

A working set of data blocks to be flushed from a cache to non-volatile data storage is identified, and a physical block of non-volatile data storage is allocated. At least some of the data blocks in the working set are compressed to generate compressed data blocks. The compressed data blocks are stored into the physical block of non-volatile data storage. For each one of the compressed data blocks stored into the physical block, an independent virtual layer pointer indicating a location of that compressed data block within the physical block of non-volatile data storage is stored into a single, flexibly sized virtual layer block that is allocated for storing the working set. The flexibly sized virtual layer block is part of a logical address mapping tree.

Flexibly sized virtual layer blocks for logical address mapping

Logical address space portions and virtual layer blocks (VLBs) can be partitioned into multiple sets. Each of multiple nodes in a system can be assigned exclusive ownership of one of the multiple sets. In at least one embodiment, for a read I/O which is received at a first node and directed to a logical address LAthat is owned by a second node, the first node can request that the second owning node perform resolution processing for LA. The second node can return either a VLB address or a PLB address based on whether the second node owns a VLB used in mapping LAto a corresponding physical location PAwhich includes content Cstored at LA. The second node can set a flag in its response to indicate whether a returned address is a VLB address or a PLB address.

Read I/O processing techniques using remote mapping resolution with logical address space slicing

A method of managing end-of-life processing of data blocks includes performing a decref procedure for a chunk of data, the decref procedure decrementing reference counts for respective blocks of the chunk of data, and deferring performance of an end-of-life procedure for a plurality of blocks of the chunk of data whose reference counts become zero as a result of performing the decref procedure. The method further includes, upon performing a data consolidation procedure after performing the decref procedure, performing the end-of-life procedure on the plurality of blocks.

End-of-life data page processing

Techniques for keeping on-disk, physical deduplication ('dedupe') index levels up to date to increase the efficiency of data deduplication in storage systems. The techniques include providing a plurality of dedupe indexes across a memory level and a disk level of a storage system, in which the plurality of dedupe indexes include an in-memory dedupe index at the memory level, and an on-disk dedupe index at the disk level. The techniques include applying a plurality of eviction rules regarding evicting entries of the on-disk dedupe index, in which the plurality of eviction rules avoid random evictions of entries from the on-disk dedupe index. The techniques include, in response to applying the plurality of eviction rules, identifying a plurality of entries of the on-disk dedupe index as eviction candidates. The techniques include evicting the identified eviction candidates from the on-disk dedupe index.

Method to keep on-disk physical deduplication index layer up to date

A method, computer program product, and computing system for receiving a user data page for storing in a storage system. An unaligned hash representation of the received user data page may be generated. The generated unaligned hash representation may be identified within a hash table, thus defining an identified hash representation. A user data page associated with the identified hash representation may be identified, thus defining an identified user data page. A hash offset reference of the identified user data page may be compared with a hash offset reference of the received user data page. A deduplication operation may be performed on the received user data page based upon, at least in part, the comparison of the hash offset reference of the identified user data page with the hash offset reference of the received user data page.

System and method for performing unaligned deduplication

A technique is directed to providing data storage resiliency. The technique includes, after losing access to a storage device that provides storage to a storage extent, performing a degraded write operation that writes data to a first line of the storage extent. The technique further includes setting a first indicator to indicate that the first line was written to while access to the storage device was lost and while a second indicator continues to indicate that a second line of the storage extent was not written to. The technique further includes, after regaining access to the storage device and in response to the first indicator indicating that the first line was written to in the degraded write operation, performing a resiliency operation that constructs additional data from the data written in the first line to increase data storage resiliency of the storage extent.

Providing data storage resiliency

A method, computer program product, and computing system for maintaining a pointer quantity count for each of a plurality of virtual mapping groups, wherein each virtual mapping group is associated with a duplicative data portion within a storage platform and each pointer quantity count defines the number of pointers included within the corresponding virtual mapping group; and obtaining a hash value for a specific duplicative data portion associated with a specific virtual mapping group if a specific pointer quantity count for the specific virtual mapping group is reduced to zero.

System and method for hash-based data deduplication

Techniques coordinate deduplication among nodes. The techniques involve, in response to a lookup query to search a deduplication index for an entry that maps a fingerprint which is based on incoming data, detecting an invalid result. The techniques further involve, in response to detecting the invalid result, updating the deduplication index to include an entry that maps the fingerprint, the entry initially storing an 'in-progress' flag. The techniques further involve, after the deduplication index is updated to include the entry that maps the fingerprint and after the incoming data is stored in a storage location, removing the “in-progress” flag from the entry.

Coordinating deduplication among nodes

A method, computer program product, and computing system for expanding the lifetime of virtual entries in a data storage system. A deduplication function of a virtual layer of a data storage system is monitored, a reference count of a virtual entry is incremented when a data page is written to a physical layer of the data storage system and the reference count of the virtual entry is decremented when a data page is deleted from the virtual layer. The virtual entry in the virtual layer is maintained when the reference count reaches a predetermined value and the virtual entry is reclaimed when a predetermined action of the data storage system is to be performed.

System and method for expanding the lifetime of virtual entries in data storage systems

In at least one embodiment, processing can include receiving at a first node a read I/O operation that requests to read content stored at a logical address LA owned by a second node. Responsive to the second node receiving a request regarding LA from the first node, the second node can perform address resolution processing using mapping information for LA that includes: an address PAof a page Pof storage of a first PLB (physical large block) where Cis stored, a value Vof a generation identifier GenIDof Pand the first PLB, and a hash value HVfor C. A reply, as returned from the second node to the first node, can include information including PAand hints including Vand HV. The first node can perform processing to determine, using the information, whether Cstored at PAwas overwritten during a vulnerability window.

Techniques for providing data consistency

A method of data deduplication during log flushing includes use of an in-memory deduplication index as a cached sub-set of entries from a full on-drive deduplication index. In a pre-flush process, the data is scanned and the on-drive index is accessed to identify matching deduplication entries, and identified entries are copied into the in-memory index. The subsequent flush includes, for each data unit, (1) accessing the in-memory index to identify a matching deduplication entry for the data unit, (2) upon identifying a matching entry, storing the data unit in a deduplicated manner as a reference to previously stored data, and (3) upon not identifying a matching entry, storing the data unit in a non-deduplicated manner including a new data instance, without accessing the on-drive index. Performance and efficiency are improved by accessing only the in-memory index and avoiding the penalties of accessing the on-drive index.

Data storage system with streamlined deduplication during write log flushing

Techniques for achieving efficient removal of stale index entries from on-drive deduplication indexes using hash prefix indexing. The techniques include providing an on-drive dedupe index that includes a plurality of index entries. Each index entry includes a hash value of a data page, and an address associated with a storage location of the data page. Each index entry is assigned to a bucket defined by a respective hash prefix. The techniques include, for each data page associated with a reference count decremented to zero, logging a hash prefix and a storage address of the data page. The techniques include, for each bucket, constructing an address bag, storing, in the address bag, logged addresses whose corresponding hash prefix is the same as the respective hash prefix defining the bucket, and removing, from the bucket, each stale index entry that includes an address matching one of the addresses in the address bag.

Efficient removal of stale entries from on-drive deduplication hash tables using hash prefix indexing

System-level garbage collection is performed in an SSD-based data storage system employing log-structured writing of data. Respective reclaim unit sizes of the storage devices are identified, wherein each reclaim unit is a unit of programming and erasure used by device-level garbage collection to generate erased reclaim units for allocation for physical data storage. The data is arranged across the storage devices in first and second data units, each first data unit being of a first size U equal to the product of R and a stripe width W, each second data unit including a respective set of first data units which occupy a maximum-size reclaim unit of a storage device among those storing the first data units. The system-level garbage collection is operated with second-data-unit granularity to regularly generate empty second data units for allocation in the log-structured writing of data, reducing write amplification and device wear.

Log-structured data storage system using flexible data placement for reduced write amplification and device wear

A flush set of data pages to flush from a cache to non-volatile data storage is identified, and a compression group is identified that is made up of similar data pages within the flush set. A previously compressed data page is identified in non-volatile data storage that is similar to the data pages in the compression group. Data compression is performed on the compression group using a compression dictionary for the previously compressed data page. A flush set specific hash index and a global hash index may be used to identify the similar data pages in the compression group and the similar previously compressed data page. An indication of the similar previously compressed data page may be stored in the compressed block generated for the compression group, and the compression dictionary for the previously compressed data page need not be duplicated in the non-volatile data storage.

Synergistic data compression

In at least one embodiment, processing can include: maintaining a storage object identifier hash table (SOIDHT) that tracks storage object identifiers (IDs) that have been reassigned among storage objects and that have potential outstanding activity associated with storage objects as prior to performing corresponding reassignments; recording, in a user data (UD) log, a UD log entry corresponding to an operation directed to a storage object assigned a storage object ID, wherein the UD log entry includes a first LSN denoting a point in time when the operation is recorded in the UD log; and flushing the UD log. Responsive to determining that the storage object has a corresponding SOIDHT entry, processing can include determining whether the first LSN of the first UD log entry of the operation is less than a second LSN of the corresponding entry of the SOIDHT.

Growing community of inventors

Tel Mond, Israel

Uri Shabi