Emc IP Holding Company LLC

Dell Products Inc.

Jeremy O'Hare

Anoop Raghunathan

Rong Yu

Michael J Scharland

Stephen M Lathrop

Owen Martin

Ramesh Doddaiah

Peng Wu

Kuolin Hua

Alesia Tringale

Ken Dorman

Arieh Don

Sorin Faibish

Douglas Eric LeCrone

Alexandr Veprinsky

Adnan Sahin

Paul A Linstead

Malak Alshawabkeh

John T Fitzgerald

Xiaomei Liu

Ning Wu

Jason J Duquette

Matthew H Fredette

Mohammed Asher

Venkata Ippatapu

Benjamin A F Randolph

Steve Lathrop

David K Aha

Alexandre Lemay

Richard Goodwill

John Madden, Jr

Jiaorui Jiang

Chaitanya Lohani

Roman Gramc

Christopher W Appleby

Lionel Bella

Garret Bourke

Guy Rososhansky

A Ransomware Activity Detection System (RADS) characterizes historic read/write IO activity on a storage volume, and also characterizes historic data characteristics of the storage volume, such as the percentage reducibility of the data held in the storage volume. The RADS monitors the storage volume to identify differences between current read/write IO activity and historic read/write IO activity, as well as difference between current data characteristics of the storage volume and historic data characteristics of the storage volume. When the RADS detects a significant difference in read/write IO activity on a storage volume, that is coupled with a significant changes to the data characteristics of the storage volume, the RADS protects the storage volume and generates an alert of the possible occurrence of a ransomware attack. Protection may occur prior in connection with any bulk read operation to proactively protect storage volumes against ransomware attacks.

Method and apparatus for detecting the occurrence of a ransomware attack on a storage volume

A storage system includes storage resources, front-end devices, and back-end devices that are subdivided into allocation units. Each front-end device has a plurality of tracks, which map to entries in the allocation units. Front-end metadata is used to specify, for a respective track, a respective back-end device, an allocation unit within the respective back-end device, and an entry within the respective allocation unit where the data for the respective track is stored. Back-end metadata is used to specify, for a respective back-end entry, a respective location in the storage resources of the storage system where the data for the respective track is stored and a size of the data on the storage resources. The back-end metadata is organized using shared back-end metadata memory pages, in which each page of metadata is equally shared between two or more of the allocation units.

Direct addressing and efficient storage of variable sized metadata

A task generator runs on a data storage system in which multiple compute nodes allocate portions of local memory to a remotely accessible shared memory. The task generator is responsive to a primary task to be performed on a storage object to generate separate, per-compute node secondary tasks corresponding to the primary task. Each of the separate secondary tasks specifies at least one attribute of metadata associated with the storage object and at least one task logic function. Each of the plurality of compute nodes performs the separate secondary task generated for that compute node by scanning the local portion of the shared memory based on the attribute to identify matching pages of the metadata associated with the storage object and performs the task logic function on the identified matching pages of the metadata associated with the storage object to generate a local result. The task generator combines the local results to perform the primary task.

Scanning pages of shared memory

The meta data containing count and key fields of CKD records are reversibly decoupled from the user data of the data field so that the data can be deduplicated. Multiple CKD records may be coalesced into a larger size CKD track. The coalesced meta data is compressed and stored in a CKD hash table. The user data is hashed, and the hash is used as a hash key that is associated with the compressed meta data in the CKD hash table. When the hash of user data associated with a CKD write IO matches the hash key of an existing entry in the table, data duplication is indicated. The compressed meta data is added to the entry and the user data is deduplicated by creating storage system meta data that points to the pre-existing copy of the user data. The storage system metadata includes unique information that enables the corresponding compressed metadata to be subsequently located in the hash table to reassemble the CKD records.

Inline deduplication for CKD using hash table for CKD track meta data

Aspects of the present disclosure relate to data deduplication (dedup) techniques for storage arrays. In embodiments, a sequence of input/output (IO) operations in an IO stream received from one or more host devices by a storage array are identified. Additionally, a determination can be made as to whether a set of previously received IO operations match the identified IO sequence based on a time series relationship between the identified IO sequence and the previously received IO operations. Further, one or more data deduplication (dedup) techniques can be performed on the matching IO sequence.

Misaligned IO sequence data deduplication (dedup)

Accessing additional storage space of a storage system includes reading a physical data fragment that is an incremental subset of a physical storage unit of the storage system, accessing metadata corresponding to logical blocks stored on the physical data fragment to determine a sidebar storage portion of the physical data fragment that is unused by the logical blocks, and accessing data of the sidebar storage portion. Accessing data of the sidebar storage portion may include providing data from the sidebar storage portion to a calling process. Accessing data of the sidebar storage portion may include modifying a portion of data from the physical data fragment and writing the physical data fragment back to the physical storage device. The physical data fragment may be locked prior to reading the physical data fragment and the physical data fragment may be unlocked after writing the physical data fragment.

Accessing storage space of a physical data fragment in a storage system

A deduplication engine maintains a deduplication hash table containing a plurality of fingerprint entries identifying a track of current data and a hash value computed from the data currently stored at the identified track. A hash table manager implements a dynamic deduplication hash table management process, that includes a fingerprint entry purge process and a fingerprint entry re-entry process. The purge process is implemented on the deduplication hash table to remove selected fingerprint entries of tracks of data currently stored on the managed drives from the deduplication hash table based on a first set of Thin Device (TDev) selection rules and a second set of track selection purge rules. The re-entry process is implemented on the deduplication hash table to re-enter selected fingerprint entries into the deduplication hash table based on a third set of TDev selection rules and a fourth set of track selection re-entry rules.

Dynamic deduplication hash table management

A hardware offload includes a hash engine that performs hashing for a block-based storage system. The hash engine calculates multiple hash values for each input buffer provided by the storage system. The hash values may be calculated with variably offset and overlapping portions of the input buffer, wherein each portion is larger than the native block size of the storage system. The hardware offload may also include a compression engine that performs compression on the input buffer using the entire input buffer and/or chunks as compression domains.

Multiple overlapping hashes at variable offset in a hardware offload

A method of determining and depicting an effective storage capacity of a storage system includes determining a storage-based limit of effective storage capacity and a metadata-based limit of effective storage capacity. The storage-based limit is based on an amount of unused capacity of a set of managed drives and a data reduction ratio achieved when host data is reduced prior to storage on the set of managed drives. Data reduction may include compression, deduplication, and pattern detection operations. The metadata-based limit is based on a volume of metadata that has been generated by the data, and the data reduction operations, in connection with writing the data to the set of managed drives, and based on an amount of memory allocated to storing the metadata. The effective storage capacity, actual storage usage, and data reduction ratio are graphically depicted over time to enable changes to these parameters to be visualized.

Method and apparatus for determining and depicting effective storage capacity of a storage system

Techniques for processing data may include: receiving source physical storage allocation units that include valid data and one or more holes of unused physical storage, wherein each source physical storage allocation unit has an associated timestamp denoting a most recent destaging time of any data stored on the source physical storage allocation unit; determining an age for each source physical allocation unit based on the associated timestamp of the source physical allocation unit; for each source physical storage allocation unit, determining one of multiple age buckets for the source physical allocation unit based on the age of the source physical allocation unit; and for a first of the age buckets including two source physical allocation units, performing first processing including: relocating at least some valid data from the two source physical storage allocation units to a target physical storage allocation unit.

Defragmentation techniques

Protection and recovery is provided for an allocation unit of data that is in-flight between a source device and a target device of a storage system during a systemic failure by selecting between corresponding source device and target device metadata based on consistency and reciprocal linkage. If logical volume metadata for the allocation unit of the data in volatile memory is not consistent with logical volume metadata for the allocation unit in non-volatile storage, then the allocation unit is recovered from the source device. If target device metadata for the data in volatile memory is not consistent with target device metadata for the data in non-volatile storage, then the allocation unit is recovered from the source device. If linkage between logical volume metadata for the allocation unit and target device metadata for the data is not reciprocal, then the allocation unit is recovered from the source device. If none of those conditions are satisfied, then the allocation unit is recovered from the target device.

Efficient resilience in a metadata paging array for in-flight user data

A method and system for sharing data reduction metadata with storage systems. Specifically, the disclosed method and system entail communicating, to a storage system, information known to host devices from which data (submitted to-be-written to the storage system) may originate. This a priori reduction-pertinent information, which may include the potential to improve storage system efficiency and/or performance at least with respect to data reduction processing of the data submitted to-be-written, had previously been considered incommunicable to the storage system. The disclosed method and system, however, lift this previous limitation and enable communication of any storage system performance-improving information, applicable to the data submitted to-be-written, to the storage system.

Method and system for sharing data reduction metadata with storage systems

Within a storage array, allocation of physical storage capacity within a storage array may be managed in standard size allocation units of uncompressed data, e.g. 128kb tracks, while smaller sub-allocation unit compression domains, e.g. 32kb quarter tracks, are used for compressed data. The data within a sub-allocation unit may be compressed to a size that is less than the capacity of the sub-allocation unit. Data associated with sub-allocation units that are not required to service a read or write may not need to be compressed or decompressed in order to service the read or write. Consequently, resource usage may be more efficient.

Use of compression domains that are more granular than storage allocation units

The system, devices, and methods disclosed herein relate to data ratio reduction technology adapted to reduce storage costs by weeding out duplicative data write operations. The techniques and systems disclosed achieve deduplication benefits of smaller block, finer granularity, deduplication systems while simultaneously reducing the enormous metadata costs traditionally associated with small block deduplication. According to an exemplary 128 K block deduplication system and method, we disclose deduplication rates nearly equivalent to 4 K deduplication systems.

Large block misaligned deduplication

The system, devices, and methods disclosed herein relate to data ratio reduction technology adapted to reduce storage costs by weeding out duplicative data write operations. The techniques and systems disclosed achieve deduplication benefits by reducing the size of hash values stored hash tables used to compare unwritten data blocks to data that has already been written and stored somewhere in physical storage. The data deduplication systems, methods, and products facilitate deduplication at the block level as well as for misaligned data chunks within data blocks, that is an unwritten data block that has been stored sequentially in two different physical locations. The deduplication teachings herein are amenable to varying data block sizes as well as data chunk sizes within blocks. Our embodiments enhance computer performance by substantially reducing computational speeds and storage requirements attendant to deduplication systems using larger hash table data sizes.

Growing community of inventors

Westborough, MA, United States of America

Jeremy O'Hare