Dell Products Inc.

Emc IP Holding Company LLC

Dell & Products L.p.

John Creed

Owen Martin

Andrew Chanler

John Krasner

Arieh Don

Ramesh Doddaiah

Kaustubh S Sahasrabudhe

Kaustubh Sahasrabudhe

Evan Barry

Steve Ivester

A method of intelligently recognizing faulty component error patterns to prevent cascading errors from causing indictment of healthy hardware components includes a system hardware error manager with a trained error analysis engine. The error analysis engine is trained using labeled training examples correlating storage system error patterns with component indictments and unindictable dependent components. The trained error analysis engine is deployed to monitor sequences of error messages to recognize error patterns generated by components of an operating storage system. In response to recognition of an error pattern, the trained error analysis engine indicts a system component associated with the recognized error pattern. Any errors that were generated by dependent components after the start of the recognized error pattern are reversed. Any dependent components that were indicted based on errors that were generated by the dependent components after the start of the recognized error pattern are also reversed.

Intelligent faulty component error pattern recognition and fault isolation

A data storage node includes a plurality of compute nodes that allocate portions of local memory to a shared cache. The shared cache is configured with mirrored and non-mirrored segments. The mirrored and non-mirrored segments are separately configured with pools of data slots. Within each segment, each pool is associated with same-size data slots that differ in size relative to the data slots of other pools. The sizes of the pools in each segment are adjusted to minimize cache loss, where cache loss is the difference between the amount of shared cache used to service IOs and the minimum amount required for servicing the IOs.

Using cache loss signal as a basis to optimize hit rate and utilization through cache partitioning

One or more aspects of the present disclosure relate to cache layout optimization. In embodiments, an input/output (IO) workload is received by a storage array. Additionally, a system-level cache layout distribution model is established based on one or more characteristics of the IO workload. Further, cache slot allocations are dynamically adjusted for each cache segment of global memory for each board based on the system-level cache layout distribution model and the one or more characteristics of the IO workload.

Board level dynamic cache layout distribution

A storage system is configured with pools of processor cores. Each pool corresponds uniquely to one of the supported service levels of the storage system. In a dynamic time series forecast adjustment mode, processor cores within each pool run at a clock speed that is statically defined for the service level corresponding to the respective pool and core affiliations with pools are dynamically adjusted based on modelled data access latency. During a scheduled event, an event guided core matrix profile overrides the time series forecast adjusted core matrix profile. The event guided core matrix profile includes core clock speeds and pool affiliations, thereby enabling rapid reconfiguration.

Customer informed composable core matrix for sustainable service levels

An online system for update compatibility verification is provided, that enables the compatibility of a proposed update to be verified against a current storage system configuration prior to be implemented on the storage system. Updates, in this context, include hardware updates, firmware updates, software updates, and combinations of updates. In connection with initiating implementation of an update on a storage system, the current storage system configuration information is used by the update compatibility verification system to determine whether the update has been identified as having caused errors or performance degradations in other similarly configured storage systems, and hence is not compatible with the storage system based on the current storage system configuration. In instances where the update compatibility verification system determines that the update is not compatible with the current storage system configuration, the update is prohibited from being applied to the storage system.

Online update compatibility verification prior to update implementation

Feature usage profiles of shared global memory are monitored and the profiles are used to allocate computational resources (CPU cores) between thread groups. A global memory monitor is used to monitor global memory metadata to determine the amount of space currently being consumed by a particular feature or service, and dynamically adjusts the amount of computational resources allocated to the thread group responsible for processing IO operations related to the feature or service. By exploiting an inversely proportional relationship between allocated computational resources and usage of shared global memory, it is possible to monitor feature usage profiles of shared global memory to dynamically allocate computational resources to thread groups. By looking at anomalies in usage of slots of shared global memory, and in a targeted way speeding up the processes working on such slots, it is possible to improve overall utilization of shared global memory.

Method and apparatus for dynamic CPU allocation based on feature usage profiles of shared global memory

One or more aspects of the present disclosure relate to isolating faults corresponding to a thread or thread group to avoid dropping a director processor slice or board. In embodiments, an execution flow corresponding to a slice on a director board of a storage array is monitored. For example, the execution flow includes a thread or a thread group. Additionally, an exception event corresponding to the execution flow is identified. Further, the execution flow is isolated based on an analysis of the exception event.

Fault isolation in a director processor slice or board

Operational information in a storage system is collected regarding storage media storage tiers, devices, drives, tracks on drives, and logical storage layers, to determine an estimated amount of time it will take to write data from cache to the intended drive when a new write operation arrives at the storage system. This information is then used to decide which type of cache is most optimal to store the data for the write operation, based on the estimated amount of time it will take to write data out from the cache. By allocating cache slots from a faster cache to write operations that are expected to quickly be written out to memory, and allocating cache slots from the slower cache to write operations that are expected to take more time to be written out to memory, it is possible to increase the availability of the cache slots in the faster cache.

Method and apparatus for cache slot allocation based on data origination location or final data destination location

A data verification process is implemented before performing role reversal on a remote data replication facility to identify data mismatch errors prior to failover or failback on the remote data replication facility. In situations where the role reversal is planned sufficiently far in advance, a full data scan is implemented by comparing the Data Integrity Field (DIF) information of each track of data on the primary storage array with the DIF information of each corresponding track of data on the backup storage array. In situations where the role reversal is more imminent, a quick scan is implemented by comparing metadata signatures for each track of data on the primary storage array with the metadata signatures of each corresponding track of data on the backup storage array. Once any data mismatch errors are identified, the data on the backup storage array can be corrected prior to role reversal.

Data error identification prior to role reversal on a remote data replication facility

High priority host applications are protected from data loss associated with failure of multiple drives associated with the same protection group by preemptively relocating associated host application image data to protections groups that do not have failed drives. New storage allocations for the high priority host applications are preferentially located on protections groups that do not have failed drives. Following replacement of the failed drive, the preemptively relocated data and new allocations are redistributed across protection groups in a balanced manner. Protection groups are characterized as targets, sources, or neutral based on allocations of high priority host application image data relative to other protection groups. Sources and paired with targets and portions of the high priority host application image data are iteratively relocated until all protection groups are neutral.

Self-healing service level storage

A multi-node storage system uses local mutual exclusion to perform fabricless local memory access and non-local mutual exclusion to perform memory access via an inter-nodal fabric. A local access mode bit and remote access request flag are associated with each data slot and its associated metadata in shared memory of the storage system. Local mutual exclusion such as a Peterson Lock is used to avoid conflict between local IO threads when the local access mode is set and the remote access request flag is not set. A remote IO thread that determines that the local access mode bit for a target slot is set will set the remote access flag. In response, a local IO thread that obtains the Peterson Lock will unset the local access mode bit and release the Peterson Lock, thereby enabling remote access to the slot and its metadata by the remote IO thread. The remote access request flag is cleared and the local access mode enabled after the remote access is completed.

Atomic CPU access to shared metadata in a fabric-attached multi-node array with shared memory

In a data storage system that includes two or more storage engines in interconnected by a channel-based fabric, highly deduplicated data is identified and promoted to a dedupe performance tier characterized by mirrors on each of the storage engines. Blocks of deduplication data are scored based on reference counts and read-miss access patterns. Promotion of blocks to the dedupe performance tier and demotion of blocks from the dedupe performance tier are calculated based on the scores. Performance is enhanced by enabling highly deduplicated blocks to be accessed from engine-local memory and engine-local storage via switches rather than the channel-based fabric.

Optimizing protection of highly deduped data for performance and availability

The cause of a failure to satisfy a high priority service level objective of a storage object or storage group is localized within a storage array. Storage objects that have been assigned low priority service level objectives are analyzed to determine whether their performance paths overlap with the performance path of the high priority service level objective storage object or storage group at the location of the cause of the failure. The low priority service level objective storage objects having performance paths that overlap with the performance path of the high priority service level objective storage object or storage group at the location of the cause of the failure are targeted for IO data rate reduction in order to free resources to restore compliance with the high priority service level objective. The other low priority service level objective storage objects are not targeted.

Targeted service level response time management

One or more aspects of the present disclosure relate to cache memory management. In embodiments, a global memory of a storage array into one or more cache partitions based on an anticipated activity of one or more input/output (IO) service level (SL) workload volumes can be dynamically partitioned.

Cache memory management

A data storage system with interconnected compute nodes includes a shared memory with volatile and non-volatile portions. Data tracks evicted from the volatile portion are moved to the non-volatile portion based on a cache-miss interarrival rate threshold that is calculated based on capacity and fall-through time of the non-volatile portion of the shared memory. Data extents on non-volatile drives are characterized based on dominant modes of extent-level cache-miss interarrival histograms generated using countdown timers of most recent backend accesses of sub-extents. The dominant mode of the extent in which a backend track evicted from the volatile portion of the shared memory resides is compared with the threshold in order to determine whether to move the backend track to the non-volatile portion of the shared memory.

Growing community of inventors

Innishannon, Ireland

John Creed