Dell Products Inc.

Emc IP Holding Company LLC

Other

Maor Rahamim

Uri Shabi

Vamsi K Vankamamidi

Alexander Shknevsky

Bar Harel

Oran Baruch

Vladimir Shveidel

Ronen Gazit

Zvi Schneider

Oran Asher Baruch

Alex Soukhman

Amit Zaitman

David Ben-David

Kurt W Everson

Techniques of the present disclosure can include: identifying blocks of storage available for allocation; generating a list denoting an allocation order of storage chunks of the blocks; receiving a write I/O operation that writes data to a first logical address; allocating a storage chunk in accordance with the allocation order of the list, wherein a first block includes the storage chunk and a second storage chunk; storing the first data in the storage chunk of the first block; removing the second storage chunk from the list; and creating a mapping between the first logical address and the first block indicating the second storage chunk is reserved for storing content written to a logical address included in a volume logical address range comprising the first logical address. The allocation order can spread allocation distance between blocks and chunks of the same block to avoid contention during flushes.

Storage allocation techniques for large writes using list formed from each block

Techniques can include: recording, in a log, log entries for write operations writing data pages to logical addresses; and flushing the log entries from the log, wherein flushing includes performing deduplication of the data pages. Flushing can include: partitioning the data pages into i) a first portion of the data pages that are determined to be duplicates of existing data pages, and ii) a second portion of the data pages determined to be unique data pages; atomically incrementing reference counts each corresponding to one of the existing data pages determined as identical to a data page of the first portion; and atomically binding the logical addresses to the data pages, including: binding the first portion of data pages to a corresponding third portion of the logical addresses; and binding the second portion of data pages to a corresponding fourth portion of the logical addresses.

Techniques for efficient flushing using opportunistic deduplication

For a flush request for a first block of user data directly stored into a physical block of non-volatile data storage, allocating a virtual layer block, and storing, in the virtual layer block, a first set of virtual block pointers indicating pages of the first block within the physical block. Pointers to the virtual block pointers in the first set are then stored into a leaf layer of a logical address mapping tree. The virtual layer block is identified for a subsequent flush request issued for a second block of user data directly stored into the physical block, and a second set of virtual block pointers indicating pages of the second block within the physical block is stored in the virtual layer block. Pointers to the virtual block pointers in the second set are stored into a leaf layer node of another logical address mapping tree.

Flushing directly ingested user data

Techniques for limiting or reducing storage accessibility 'hiccups' in active-active clustered systems that perform large writes. The techniques can include executing, by a surviving, failover, or rebooted node of an active-active clustered system, a specialized recovery protocol that includes treating each large write request from a host computer as a plurality of small write requests while execution of the specialized recovery protocol is in progress, draining all dedicated sub-ubers for a primary and secondary node of the active-active clustered system, and, having completed execution of the specialized recovery protocol, resuming normal treatment of large write requests from the host computer. In this way, the need and complexity of managing large writes and maintaining their corresponding sub-uber information during recovery from a forced reboot, crash, or disaster involving the primary or secondary node can be avoided, and storage accessibility hiccups due to performing the large writes can be limited or reduced.

Low hiccup time fail-back in active-active dual-node storage systems with large writes

A method, computer program product, and computing system for generating a plurality of page buffers from a log memory system of the storage system using a persistent memory organization policy. A plurality of physical layer blocks (PLBs) are generated from the log memory system using the persistent memory organization policy. A set of page buffers are allocated from the plurality of page buffers to a first mapping page of a plurality of mapping pages. A PLB is allocated from the plurality of PLBs to a second mapping page of the plurality of mapping pages. One or more IO requests are processed using one or more of the first mapping page and the second mapping page.

System and method for flushing data using mapping page ownership

Storage allocation techniques for large writes based on reserving one or more additional chunks of a block

A method, computer program product, and computer system for identifying, by a computing device, content in a first bucket in a first cache. It may be determined that a first portion of the content in the first bucket is a duplicate, wherein a second portion of the content in the first bucket may be unique. The first portion of the content in the first bucket may be deduplicated from the first cache. The second portion of the content may be stored in a second bucket in a second cache.

System and method for efficient background deduplication during hardening

An aspect of implementing amortized execution of updates for a hash table includes aggregating, within a data structure, updates to be executed for the hash table. The aggregated updates are distributed across a plurality of chunks in the data structure. An aspect also includes sorting, within each of the chunks, the updates according to respective bucket index values associated with the updates, and iteratively executing the sorted updates across each of the chunks in the data structure.

Amortized execution of updates to a hardened hash table

Techniques for rebuilding data in a data storage system are provided. A method includes: (a) identifying (i) a first set of degraded Ubers that contain no portions reserved for direct writing and (ii) a second set of degraded Ubers that contain at least one portion reserved for direct writing. Direct writing is a process that writes blocks to long-term storage prior to mapping those blocks in a metadata mapping structure. An Uber is a set of adjacent stripes across a respective Redundant Array of Independent Disks (RAID) array of the data storage system, and a degraded Uber is an Uber that includes at least one failed drive within its RAID array. The method further includes (b) initiating a rebuild of the first set of degraded Ubers; and (c) delaying a rebuild of each degraded Uber of the second set until all pending direct writes to blocks of that degraded Uber have been mapped by the metadata mapping structure.

Techniques for performing live rebuild in storage systems that operate a direct write mode

An aspect includes splitting a table of buckets into a fixed number of domains. Each of the domains includes a corresponding subset of the buckets. An aspect also includes providing a spare bucket for each of the subsets of the buckets and providing a metadata structure for each of the domains. The metadata structure includes a head pointer that points to a first bucket of a corresponding subset of the buckets and a spare_bucket pointer that points to the spare bucket of the subset of the buckets. An aspect further includes providing a split-spare bucket pointer that interleaves, during updates to data, among the subset of buckets in the domain. Data subject to the updates is stored in the spare bucket for a corresponding one of the domains. An aspect also includes updating the head pointer and the spare_bucket pointer for corresponding domains in response to updating the data.

Managing hash tables in a storage system

A method, computer program product, and computer system for storing data in a bucket of a plurality of buckets. A spare bucket may be reserved in the plurality of buckets. A copy of the data may be stored in the spare bucket. A pointer to the data in the bucket and a pointer to the copy of the data in the spare bucket may be updated based upon, at least in part, storing the data in the bucket and storing the copy of the data in the spare bucket.

System and method for scalable and space efficient hardening of a fixed sized hash table over an unreliable tier

A method of performing deduplication is provided. The method includes (a) selecting, by applying a deterministic selection criterion, a sub-block of a block of data that contains multiple sub-blocks; (b) performing a lookup, into a deduplication table, of a digest generated by hashing the selected sub-block, the lookup matching an entry indexed by the digest in the deduplication table, the entry identifying a previously processed block; and (c) effecting storage of the block, including pointing to the previously processed block. An apparatus, system, and computer program product for performing a similar method are also provided.

Sub-block deduplication using sector hashing

An apparatus in one embodiment comprises at least one processing device. The processing device is configured to generate a hash entry of a hash table implementing a key-value store that comprises a plurality of buckets associated with corresponding keys where a given bucket comprises a bucket index having two or more entries for storage of values that hash to a key associated with the given bucket. The processing device is also configured to allocate blocks in non-volatile memory, to split the given bucket into chunks each comprising at least one entry of the bucket index, and to persist the hash table to the non-volatile memory utilizing a direct mapping of the chunks of the given bucket to respective designated block offset positions in the allocated blocks. Persisting the hash table comprises storing first and second chunks of the given bucket in first and second allocated blocks.

Processing device configured for efficient generation of a direct mapped hash table persisted to non-volatile block memory

A method of performing deduplication by a computing device is provided. The method includes (a) as data is received by the computing device into blocks as part of write requests, creating an entry in a log for each of the blocks, each entry including information about that respective block and a digest computed from that respective block; and (b) after accumulating multiple entries in the log, processing the log for delayed deduplication, the processing including (i) retrieving digests from the log, (ii) performing lookups within a deduplication table of the retrieved digests, and (iii) performing deduplication operations based on the lookups using the information about blocks included within the log. An apparatus, system, and computer program product for performing a similar method are also provided.

Delayed deduplication using precalculated hashes

An apparatus in one embodiment comprises at least one processing device comprising a processor coupled to a memory. The processing device is configured, in conjunction with an online cluster expansion process for a clustered storage system having an address space distributed across a set of storage controllers each associated with one or more storage devices, to modify the distribution of the address space across the set of storage controllers responsive to adding at least one of compute and storage capacity to the set of storage controllers of the clustered storage system. The processing device is also configured to identify a set of pending input-output requests directed to deletion of data in the clustered storage system, and to execute the set of pending input-output requests prior to relocating data in the clustered storage system based at least in part on the modified distribution of the address space.

Growing community of inventors

Ramla, Israel

Maor Rahamim