Emc IP Holding Company LLC

Dell Products Inc.

Other

Uri Shabi

Ronen Gazit

Vladimir Shveidel

Alex Soukhman

Maor Rahamim

Bar Harel

Anton Kucherov

Amitai Alkalay

Alexei Kabishcer

Geng Han

Vamsi K Vankamamidi

Zvi Schneider

Bar David

Alon Titelman

Nir Bar Joseph

Amit Zaitman

Dror Zalstein

Alexander Shknevsky

Xiangping Chen

Steven A Morley

Shaul Dar

Nimrod Shani

Tal Ben-Moshe

Samuel L Mullis, Ii

Michael Litvak

Jonathan Volij

Liang Huang

Aleksey Kabishcer

Rachel Fraenkel-Saban

Boaz Sela

Aleksey Kabishcher

Techniques coordinate deduplication among nodes. The techniques involve, in response to a lookup query to search a deduplication index for an entry that maps a fingerprint which is based on incoming data, detecting an invalid result. The techniques further involve, in response to detecting the invalid result, updating the deduplication index to include an entry that maps the fingerprint, the entry initially storing an 'in-progress' flag. The techniques further involve, after the deduplication index is updated to include the entry that maps the fingerprint and after the incoming data is stored in a storage location, removing the “in-progress” flag from the entry.

Coordinating deduplication among nodes

A method, computer program product, and computing system for expanding the lifetime of virtual entries in a data storage system. A deduplication function of a virtual layer of a data storage system is monitored, a reference count of a virtual entry is incremented when a data page is written to a physical layer of the data storage system and the reference count of the virtual entry is decremented when a data page is deleted from the virtual layer. The virtual entry in the virtual layer is maintained when the reference count reaches a predetermined value and the virtual entry is reclaimed when a predetermined action of the data storage system is to be performed.

System and method for expanding the lifetime of virtual entries in data storage systems

In at least one embodiment, processing can include receiving at a first node a read I/O operation that requests to read content stored at a logical address LA owned by a second node. Responsive to the second node receiving a request regarding LA from the first node, the second node can perform address resolution processing using mapping information for LA that includes: an address PAof a page Pof storage of a first PLB (physical large block) where Cis stored, a value Vof a generation identifier GenIDof Pand the first PLB, and a hash value HVfor C. A reply, as returned from the second node to the first node, can include information including PAand hints including Vand HV. The first node can perform processing to determine, using the information, whether Cstored at PAwas overwritten during a vulnerability window.

Techniques for providing data consistency

A method of data deduplication during log flushing includes use of an in-memory deduplication index as a cached sub-set of entries from a full on-drive deduplication index. In a pre-flush process, the data is scanned and the on-drive index is accessed to identify matching deduplication entries, and identified entries are copied into the in-memory index. The subsequent flush includes, for each data unit, (1) accessing the in-memory index to identify a matching deduplication entry for the data unit, (2) upon identifying a matching entry, storing the data unit in a deduplicated manner as a reference to previously stored data, and (3) upon not identifying a matching entry, storing the data unit in a non-deduplicated manner including a new data instance, without accessing the on-drive index. Performance and efficiency are improved by accessing only the in-memory index and avoiding the penalties of accessing the on-drive index.

Data storage system with streamlined deduplication during write log flushing

Techniques for achieving efficient removal of stale index entries from on-drive deduplication indexes using hash prefix indexing. The techniques include providing an on-drive dedupe index that includes a plurality of index entries. Each index entry includes a hash value of a data page, and an address associated with a storage location of the data page. Each index entry is assigned to a bucket defined by a respective hash prefix. The techniques include, for each data page associated with a reference count decremented to zero, logging a hash prefix and a storage address of the data page. The techniques include, for each bucket, constructing an address bag, storing, in the address bag, logged addresses whose corresponding hash prefix is the same as the respective hash prefix defining the bucket, and removing, from the bucket, each stale index entry that includes an address matching one of the addresses in the address bag.

Efficient removal of stale entries from on-drive deduplication hash tables using hash prefix indexing

System-level garbage collection is performed in an SSD-based data storage system employing log-structured writing of data. Respective reclaim unit sizes of the storage devices are identified, wherein each reclaim unit is a unit of programming and erasure used by device-level garbage collection to generate erased reclaim units for allocation for physical data storage. The data is arranged across the storage devices in first and second data units, each first data unit being of a first size U equal to the product of R and a stripe width W, each second data unit including a respective set of first data units which occupy a maximum-size reclaim unit of a storage device among those storing the first data units. The system-level garbage collection is operated with second-data-unit granularity to regularly generate empty second data units for allocation in the log-structured writing of data, reducing write amplification and device wear.

Log-structured data storage system using flexible data placement for reduced write amplification and device wear

A flush set of data pages to flush from a cache to non-volatile data storage is identified, and a compression group is identified that is made up of similar data pages within the flush set. A previously compressed data page is identified in non-volatile data storage that is similar to the data pages in the compression group. Data compression is performed on the compression group using a compression dictionary for the previously compressed data page. A flush set specific hash index and a global hash index may be used to identify the similar data pages in the compression group and the similar previously compressed data page. An indication of the similar previously compressed data page may be stored in the compressed block generated for the compression group, and the compression dictionary for the previously compressed data page need not be duplicated in the non-volatile data storage.

Synergistic data compression

In at least one embodiment, processing can include: maintaining a storage object identifier hash table (SOIDHT) that tracks storage object identifiers (IDs) that have been reassigned among storage objects and that have potential outstanding activity associated with storage objects as prior to performing corresponding reassignments; recording, in a user data (UD) log, a UD log entry corresponding to an operation directed to a storage object assigned a storage object ID, wherein the UD log entry includes a first LSN denoting a point in time when the operation is recorded in the UD log; and flushing the UD log. Responsive to determining that the storage object has a corresponding SOIDHT entry, processing can include determining whether the first LSN of the first UD log entry of the operation is less than a second LSN of the corresponding entry of the SOIDHT.

Storage space accounting techniques allowing storage object identifier reuse and reassignment

In at least one embodiment, processing can include receiving, at a first node, a read request directed to a logical address LA owned by a second node. The first node can locally cache content and an address hint corresponding to LA. The first node can issue a request to the second node. The request can include a flag to suppress the second node from returning content stored at the target logical address. The first node can receive a response including an address used to read current content of LA1 from back-end non-volatile storage. The first node can determine whether the address matches the address hint cached on the first node. If the first node determines the address and address hint match, the cached content of LA stored on the first node is valid and can be returned in response to the read as current content stored at LA.

Data caching strategies for storage with ownership of logical address slices

Logical address slices and corresponding metadata pages of mapping information can be partitioned into sets. Each node can be assigned exclusive ownership of one of the sets. In at least one embodiment, for a read I/O which is received at a first node and directed to a logical address LAthat is owned by a second node, the first node can request that the second owning node perform resolution processing for LA. The second node can own both LAand corresponding metadata pages included in mapping information used to map LAto a corresponding physical location PAincluding content Cstored at LA. The second node can perform resolution processing for LAusing the metadata pages corresponding to LAto either read and return Cto the first node, or obtain and return PAto the first node where the first node can then read Cdirectly using PA

Exclusive ownership of logical address slices and associated metadata

Techniques for providing a 'bin-less' metadata page allocator for clustered systems with log-structured metadata storage. The techniques include providing a mapping structure in memory of a storage node of a clustered system. The mapping structure can have multiple layers configured to map logical addresses of metadata pages to physical addresses of storage drives within a storage array. The techniques include providing a translation table configured to translate logical addresses of metadata pages to corresponding current physical addresses of storage drives within the storage array. The techniques include, in response to an allocated logical address of a metadata page no longer being in-use, replacing its corresponding current physical address with a predefined value in the translation table, freeing the logical address, and inserting the freed logical address into a free logical address array. The techniques include allocating the freed logical address from the free logical address array.

Bin-less metadata pages allocator for storage cluster with log-structured metadata

Techniques described herein elect how data is deduplicated in a storage system. A similarity hash signature for a data unit is calculated. A digest table is searched for a similarity hash signature within a predetermined distance of the similarity hash signature for the data unit. Based on the search, either a similarity hash signature or a strong hash signature of the data unit is added to the digest table.

Elective deduplication

Logical address space portions and virtual layer blocks (VLBs) can be partitioned into multiple sets. Each of multiple nodes in a system can be assigned exclusive ownership of one of the multiple sets. In at least one embodiment, for a read I/O which is received at a first node and directed to a logical address LAthat is owned by a second node, the first node can request that the second owning node perform resolution processing for LA. The second node can return either a VLB address or a PLB address based on whether the second node owns a VLB used in mapping LAto a corresponding physical location PAwhich includes content Cstored at LA. The second node can set a flag in its response to indicate whether a returned address is a VLB address or a PLB address.

Read I/O processing techniques using remote mapping resolution with logical address space slicing

A method is used for improving performance in a storage system during deduplication. A processor receives a request to write data. A determination is made that the data can be deduplicated. A non-exclusive lock for a page in storage with a reference count of the data may be obtained, and an instruction to increment the reference count for the data may be stored in a metadata journal. The non-exclusive lock for the page may be released.

Method and system for improving performance during deduplication

A technique of updating leaf pointers in a storage system includes creating a point-in-time copy of persisted metadata pages and scanning the copy for leaf pointers that point to redirected virtual blocks. Metadata pages that contain leaf pointers that point to redirected virtual blocks are read from cache and processed for updates, but no action is taken in cache for metadata pages that do not contain such leaf pointers.

Growing community of inventors

Tel Mond, Israel

Uri Shabi