Emc IP Holding Company LLC

Dell Products Inc.

Other

Uri Shabi

Ronen Gazit

Vladimir Shveidel

Alex Soukhman

Maor Rahamim

Bar Harel

Anton Kucherov

Amitai Alkalay

Alexei Kabishcer

Geng Han

Zvi Schneider

Vamsi K Vankamamidi

Bar David

Alon Titelman

Nir Bar Joseph

Amit Zaitman

Dror Zalstein

Xiangping Chen

Steven A Morley

Shaul Dar

Nimrod Shani

Tal Ben-Moshe

Samuel L Mullis, Ii

Michael Litvak

Alexander Shknevsky

Jonathan Volij

Liang Huang

Aleksey Kabishcer

Aleksey Kabishcher

A method of data deduplication during log flushing includes use of an in-memory deduplication index as a cached sub-set of entries from a full on-drive deduplication index. In a pre-flush process, the data is scanned and the on-drive index is accessed to identify matching deduplication entries, and identified entries are copied into the in-memory index. The subsequent flush includes, for each data unit, (1) accessing the in-memory index to identify a matching deduplication entry for the data unit, (2) upon identifying a matching entry, storing the data unit in a deduplicated manner as a reference to previously stored data, and (3) upon not identifying a matching entry, storing the data unit in a non-deduplicated manner including a new data instance, without accessing the on-drive index. Performance and efficiency are improved by accessing only the in-memory index and avoiding the penalties of accessing the on-drive index.

Data storage system with streamlined deduplication during write log flushing

Techniques for achieving efficient removal of stale index entries from on-drive deduplication indexes using hash prefix indexing. The techniques include providing an on-drive dedupe index that includes a plurality of index entries. Each index entry includes a hash value of a data page, and an address associated with a storage location of the data page. Each index entry is assigned to a bucket defined by a respective hash prefix. The techniques include, for each data page associated with a reference count decremented to zero, logging a hash prefix and a storage address of the data page. The techniques include, for each bucket, constructing an address bag, storing, in the address bag, logged addresses whose corresponding hash prefix is the same as the respective hash prefix defining the bucket, and removing, from the bucket, each stale index entry that includes an address matching one of the addresses in the address bag.

Efficient removal of stale entries from on-drive deduplication hash tables using hash prefix indexing

System-level garbage collection is performed in an SSD-based data storage system employing log-structured writing of data. Respective reclaim unit sizes of the storage devices are identified, wherein each reclaim unit is a unit of programming and erasure used by device-level garbage collection to generate erased reclaim units for allocation for physical data storage. The data is arranged across the storage devices in first and second data units, each first data unit being of a first size U equal to the product of R and a stripe width W, each second data unit including a respective set of first data units which occupy a maximum-size reclaim unit of a storage device among those storing the first data units. The system-level garbage collection is operated with second-data-unit granularity to regularly generate empty second data units for allocation in the log-structured writing of data, reducing write amplification and device wear.

Log-structured data storage system using flexible data placement for reduced write amplification and device wear

A flush set of data pages to flush from a cache to non-volatile data storage is identified, and a compression group is identified that is made up of similar data pages within the flush set. A previously compressed data page is identified in non-volatile data storage that is similar to the data pages in the compression group. Data compression is performed on the compression group using a compression dictionary for the previously compressed data page. A flush set specific hash index and a global hash index may be used to identify the similar data pages in the compression group and the similar previously compressed data page. An indication of the similar previously compressed data page may be stored in the compressed block generated for the compression group, and the compression dictionary for the previously compressed data page need not be duplicated in the non-volatile data storage.

Synergistic data compression

In at least one embodiment, processing can include: maintaining a storage object identifier hash table (SOIDHT) that tracks storage object identifiers (IDs) that have been reassigned among storage objects and that have potential outstanding activity associated with storage objects as prior to performing corresponding reassignments; recording, in a user data (UD) log, a UD log entry corresponding to an operation directed to a storage object assigned a storage object ID, wherein the UD log entry includes a first LSN denoting a point in time when the operation is recorded in the UD log; and flushing the UD log. Responsive to determining that the storage object has a corresponding SOIDHT entry, processing can include determining whether the first LSN of the first UD log entry of the operation is less than a second LSN of the corresponding entry of the SOIDHT.

Storage space accounting techniques allowing storage object identifier reuse and reassignment

In at least one embodiment, processing can include receiving, at a first node, a read request directed to a logical address LA owned by a second node. The first node can locally cache content and an address hint corresponding to LA. The first node can issue a request to the second node. The request can include a flag to suppress the second node from returning content stored at the target logical address. The first node can receive a response including an address used to read current content of LA1 from back-end non-volatile storage. The first node can determine whether the address matches the address hint cached on the first node. If the first node determines the address and address hint match, the cached content of LA stored on the first node is valid and can be returned in response to the read as current content stored at LA.

Data caching strategies for storage with ownership of logical address slices

Logical address slices and corresponding metadata pages of mapping information can be partitioned into sets. Each node can be assigned exclusive ownership of one of the sets. In at least one embodiment, for a read I/O which is received at a first node and directed to a logical address LAthat is owned by a second node, the first node can request that the second owning node perform resolution processing for LA. The second node can own both LAand corresponding metadata pages included in mapping information used to map LAto a corresponding physical location PAincluding content Cstored at LA. The second node can perform resolution processing for LAusing the metadata pages corresponding to LAto either read and return Cto the first node, or obtain and return PAto the first node where the first node can then read Cdirectly using PA

Exclusive ownership of logical address slices and associated metadata

Techniques for providing a 'bin-less' metadata page allocator for clustered systems with log-structured metadata storage. The techniques include providing a mapping structure in memory of a storage node of a clustered system. The mapping structure can have multiple layers configured to map logical addresses of metadata pages to physical addresses of storage drives within a storage array. The techniques include providing a translation table configured to translate logical addresses of metadata pages to corresponding current physical addresses of storage drives within the storage array. The techniques include, in response to an allocated logical address of a metadata page no longer being in-use, replacing its corresponding current physical address with a predefined value in the translation table, freeing the logical address, and inserting the freed logical address into a free logical address array. The techniques include allocating the freed logical address from the free logical address array.

Bin-less metadata pages allocator for storage cluster with log-structured metadata

Techniques described herein elect how data is deduplicated in a storage system. A similarity hash signature for a data unit is calculated. A digest table is searched for a similarity hash signature within a predetermined distance of the similarity hash signature for the data unit. Based on the search, either a similarity hash signature or a strong hash signature of the data unit is added to the digest table.

Elective deduplication

Logical address space portions and virtual layer blocks (VLBs) can be partitioned into multiple sets. Each of multiple nodes in a system can be assigned exclusive ownership of one of the multiple sets. In at least one embodiment, for a read I/O which is received at a first node and directed to a logical address LAthat is owned by a second node, the first node can request that the second owning node perform resolution processing for LA. The second node can return either a VLB address or a PLB address based on whether the second node owns a VLB used in mapping LAto a corresponding physical location PAwhich includes content Cstored at LA. The second node can set a flag in its response to indicate whether a returned address is a VLB address or a PLB address.

Read I/O processing techniques using remote mapping resolution with logical address space slicing

A method is used for improving performance in a storage system during deduplication. A processor receives a request to write data. A determination is made that the data can be deduplicated. A non-exclusive lock for a page in storage with a reference count of the data may be obtained, and an instruction to increment the reference count for the data may be stored in a metadata journal. The non-exclusive lock for the page may be released.

Method and system for improving performance during deduplication

A technique of updating leaf pointers in a storage system includes creating a point-in-time copy of persisted metadata pages and scanning the copy for leaf pointers that point to redirected virtual blocks. Metadata pages that contain leaf pointers that point to redirected virtual blocks are read from cache and processed for updates, but no action is taken in cache for metadata pages that do not contain such leaf pointers.

Adjusting leaf pointers with reduced cache usage

A method, computer program product, and computing system for loading a first chunk of data from a user data page into a first buffer. Each successive chunk of data from the user data page may be iteratively loaded into a second buffer. The first chunk of data in the first buffer may be iteratively compared with each successive chunk of data in the second buffer. In response to determining that the first chunk of data in the first buffer is equivalent to each successive chunk of data in the second buffer, the first chunk of data may be defined as a pattern.

System and method for detecting and utilizing data patterns

A request to perform a first operation in a system that stores deduplicated data can be received. The system can include a data block stored at multiple logical address each referencing the data block. A reference count can be associated with the data block and can denote a number of logical addresses referencing the data block. Processing can be performed to service the request and perform the first operation, wherein the processing can include: acquiring a non-exclusive lock for a page that includes the reference count of the data block; storing, in a metadata log while holding the non-exclusive lock on the page, an entry to decrement the reference count of the data block; and releasing the non-exclusive lock on the page.

Techniques for lock contention reduction in a log structured system

A method, computer program product, and computing system for receiving data for storage in a storage system. The data may be written to a head entry of a log buffer, wherein the log buffer includes a plurality of data entries for flushing to the storage system. At least a portion of the plurality of data entries of the log buffer may be flushed, via a plurality of threads, to the storage system based upon, at least in part, a tail entry of the log buffer. A queue of committed data entries may be updated, via each thread of the plurality of threads, with one or more data entries of the log buffer flushed to the storage system by each thread. A new tail entry of the log buffer may be determined, via a thread of the plurality of threads, based upon, at least in part, the queue of committed data entries.

Growing community of inventors

Tel Mond, Israel

Uri Shabi