Dell Products Inc.

Emc IP Holding Company LLC

Anoop Raghunathan

Jeremy O'Hare

Stephen M Lathrop

Ramesh Doddaiah

Kuolin Hua

Steve Lathrop

Owen Martin

Michael J Scharland

Rong Yu

Ning Wu

Alesia Tringale

Jason J Duquette

Venkata Ippatapu

Dustin Hunt Zentz

Vladimir Desyatov

Mark D Moreau

David K Aha

Ken Dorman

Christopher W Appleby

Garret Bourke

Chaitanya Lohani

Kanuri Venkata Satyanarayana Swami

Sean B Reilly

Lionel Bella

Roman Gramc

One or more aspects of the present disclosure relate to simulating an input/output (IO) workload. In embodiments, information corresponding to one or more characteristics of an input/output (IO) workload is received. In addition, metadata corresponding to the IO workload is generated using the information. Further, testing of a storage array can be enabled using the metadata.

Metadata simulation

A storage system includes storage resources, front-end devices, and back-end devices that are subdivided into allocation units. Each front-end device has a plurality of tracks, which map to entries in the allocation units. Front-end metadata is used to specify, for a respective track, a respective back-end device, an allocation unit within the respective back-end device, and an entry within the respective allocation unit where the data for the respective track is stored. Back-end metadata is used to specify, for a respective back-end entry, a respective location in the storage resources of the storage system where the data for the respective track is stored and a size of the data on the storage resources. The back-end metadata is organized using shared back-end metadata memory pages, in which each page of metadata is equally shared between two or more of the allocation units.

Direct addressing and efficient storage of variable sized metadata

A task generator runs on a data storage system in which multiple compute nodes allocate portions of local memory to a remotely accessible shared memory. The task generator is responsive to a primary task to be performed on a storage object to generate separate, per-compute node secondary tasks corresponding to the primary task. Each of the separate secondary tasks specifies at least one attribute of metadata associated with the storage object and at least one task logic function. Each of the plurality of compute nodes performs the separate secondary task generated for that compute node by scanning the local portion of the shared memory based on the attribute to identify matching pages of the metadata associated with the storage object and performs the task logic function on the identified matching pages of the metadata associated with the storage object to generate a local result. The task generator combines the local results to perform the primary task.

Scanning pages of shared memory

Aspects of the present disclosure relate to data deduplication (dedup) techniques for storage arrays. In embodiments, a sequence of input/output (IO) operations in an IO stream received from one or more host devices by a storage array are identified. Additionally, a determination can be made as to whether a set of previously received IO operations match the identified IO sequence based on a time series relationship between the identified IO sequence and the previously received IO operations. Further, one or more data deduplication (dedup) techniques can be performed on the matching IO sequence.

Misaligned IO sequence data deduplication (dedup)

A deduplication engine maintains a deduplication hash table containing a plurality of fingerprint entries identifying a track of current data and a hash value computed from the data currently stored at the identified track. A hash table manager implements a dynamic deduplication hash table management process, that includes a fingerprint entry purge process and a fingerprint entry re-entry process. The purge process is implemented on the deduplication hash table to remove selected fingerprint entries of tracks of data currently stored on the managed drives from the deduplication hash table based on a first set of Thin Device (TDev) selection rules and a second set of track selection purge rules. The re-entry process is implemented on the deduplication hash table to re-enter selected fingerprint entries into the deduplication hash table based on a third set of TDev selection rules and a fourth set of track selection re-entry rules.

Dynamic deduplication hash table management

In a data storage system in which a full-size allocation unit is used for storage of uncompressed data, an optimal reduced size allocation unit is selected for storage of compressed data. Changes in the compressed size of at least one full-size allocation unit of representative data are monitored over time. The representative data may be selected based on write frequency, relocation frequency, or both. Compression size values are counted and weighted to calculate the optimal reduced allocation unit size. The optimal reduced size allocation unit is used for storage of compressed data. A full-size allocation unit of data that cannot be accommodated by a reduced size allocation unit when compressed is stored uncompressed.

Efficient compressed track size classification to reduce disk fragmentation and increase probability of in-place compressed writes

A hardware offload includes a hash engine that performs hashing for a block-based storage system. The hash engine calculates multiple hash values for each input buffer provided by the storage system. The hash values may be calculated with variably offset and overlapping portions of the input buffer, wherein each portion is larger than the native block size of the storage system. The hardware offload may also include a compression engine that performs compression on the input buffer using the entire input buffer and/or chunks as compression domains.

Multiple overlapping hashes at variable offset in a hardware offload

Techniques for performing data movements may include tracking sets of I/O statistics for a extents, wherein each of the sets of I/O statistics characterizes I/O operations directed to a corresponding one of the extents having data stored on a first storage tier of first non-volatile storage devices; determining, in accordance with the sets of I/O statistics, a list that includes one or more of the extents qualified for promotion from the first storage tier to a second storage tier, wherein the second storage tier includes second non-volatile storage devices have a higher performance ranking that the first non-volatile storage devices of the first storage tier; selecting, from the list, a first extent to be promoted from the first storage tier to the second storage tier; and performing first processing that promotes only cached data of the first extent from the first storage tier to the second storage tier.

Caching and data promotion techniques

Techniques for processing data may include: receiving source physical storage allocation units that include valid data and one or more holes of unused physical storage, wherein each source physical storage allocation unit has an associated timestamp denoting a most recent destaging time of any data stored on the source physical storage allocation unit; determining an age for each source physical allocation unit based on the associated timestamp of the source physical allocation unit; for each source physical storage allocation unit, determining one of multiple age buckets for the source physical allocation unit based on the age of the source physical allocation unit; and for a first of the age buckets including two source physical allocation units, performing first processing including: relocating at least some valid data from the two source physical storage allocation units to a target physical storage allocation unit.

Defragmentation techniques

Protection and recovery is provided for an allocation unit of data that is in-flight between a source device and a target device of a storage system during a systemic failure by selecting between corresponding source device and target device metadata based on consistency and reciprocal linkage. If logical volume metadata for the allocation unit of the data in volatile memory is not consistent with logical volume metadata for the allocation unit in non-volatile storage, then the allocation unit is recovered from the source device. If target device metadata for the data in volatile memory is not consistent with target device metadata for the data in non-volatile storage, then the allocation unit is recovered from the source device. If linkage between logical volume metadata for the allocation unit and target device metadata for the data is not reciprocal, then the allocation unit is recovered from the source device. If none of those conditions are satisfied, then the allocation unit is recovered from the target device.

Efficient resilience in a metadata paging array for in-flight user data

The system, devices, and methods disclosed herein relate to data ratio reduction technology adapted to reduce storage costs by weeding out duplicative data write operations. The techniques and systems disclosed achieve deduplication benefits of smaller block, finer granularity, deduplication systems while simultaneously reducing the enormous metadata costs traditionally associated with small block deduplication. According to an exemplary 128 K block deduplication system and method, we disclose deduplication rates nearly equivalent to 4 K deduplication systems.

Large block misaligned deduplication

Allocation of physical storage capacity within a storage array may be managed in standard size allocation units, e.g. 128 kb tracks, and smaller sub-allocation units, e.g. 32 kb quarter tracks. Data associated with sub-allocation units that match a known pattern may be represented by a hint rather than data in storage. The hint may be used to mitigate overhead associated with accessing backend storage. The hint may also mitigate the need for performing compression/decompression of the corresponding sub-allocation unit if the data is compressed. In response to a read or write request associated with an allocation unit organized into sub-allocation units, only those sub-allocation units that are required to service the read or write request and fail to match a pattern are accessed, decompressed or compressed.

Growing community of inventors

Ashland, MA, United States of America

Anoop Raghunathan