Emc IP Holding Company LLC

Dell Products Inc.

Sweetesh Singh

Jonathan I Krasner

Steven R Chalmer

John Krasner

Ramesh Doddaiah

Venkata Ippatapu

Clifford Lim

Ashish Kamra

Chakib Ourraoui

A data model is trained to predict compressibility of binary data structures based on component entropy and predict relative compression efficiency for various compression algorithms based on component size. A recommendation engine in a storage system uses the data model to predict compressibility of binary data and determines whether to compress the binary data based on predicted compressibility. If the recommendation engine determines that compression of the binary data is justified, then a compression algorithm is recommended based on predicted relative compression efficiency. For example, the compression algorithm predicted to yield the greatest compression ratio or shortest compression/decompression time may be recommended.

Using machine learning to select compression algorithms for compressing binary datasets

Disclosed are techniques for compressing data in a data storage system comprising searching a cluster of nearest neighbors, wherein the cluster has been created using a locality sensitive hashing algorithm, to determine if a data block can be compressed. In alternate embodiments, nearest neighbor clusters can be formed using unsupervised learning. Additionally, nearest neighbors can also be formed in alternate embodiments using one or more of the following algorithms: a k-means clustering algorithm, a k-medoids clustering algorithm, a mean shift algorithm, a generalized method of moment (GMM) algorithm, or a density based spatial clustering of applications with noise (DBSCAN) algorithm.

Data compression using nearest neighbor cluster

Aspects of the present disclosure relate to reducing the latency of data deduplication. In embodiments, an input/output (IO) workload received by a storage array is monitored. Further, at least one IO write operation in the IO workload is identified. A space-efficient probabilistic data structure is used to determine if a director board is associated with the IO write. Additionally, the IO write operation is processed based on the determination.

Data deduplication latency reduction

A data model is trained to determine whether data is raw, compressed, and/or encrypted. The data model may also be trained to recognize which compression algorithm was used to compress data and predict compression ratios for the data using different compression algorithms. A storage system uses the data model to independently identify raw data. The raw data is grouped based on similarity of statistical features and group members are compressed with the same compression algorithm and may be encrypted after compression with the same encryption algorithm. The data model may also be used to identify sub-optimally compressed data, which may be uncompressed and grouped for compression using a different compression algorithm.

Statistical and neural network approach for data characterization to reduce storage space requirements

One or more aspects of the present disclosure relate to data protection techniques in response to power disruptions a power supply from a continuous power source for a storage device can be monitored. A power disruption event interrupting the power supply from the continuous power source can further be identified. In response to detecting an event, a storage system can be switched to a backup power supply, power consumption of one or more components of the storage device can be controlled based on information associated with each component and an amount of power available in the backup power supply. Further, one or more power interruption operations can be performed while the backup power supply includes sufficient power for performing the power interruption operations.

Power disruption data protection

Extents of data are promoted or demoted between hierarchical storage tiers based on temporal locality of access captured with a primary count-min-sketch based on monitored input-output commands. Extents may also be promoted based on spatial locality of access captured with a secondary count-min-sketch based on the monitored input-output commands. The count-min-sketches may be implemented in a tiering engine of a storage array or in the flash translation layers of solid-state drives. The flash translation layers may generate tiering hints that are provided to a tiering engine.

Locality-aware, memory-efficient, time-efficient hot data identification using count-min-sketch for flash or streaming applications

A method for processing data includes receiving a write request by a host operating system during a predetermined time window, storing data associated with the write request in a shared memory, making a first determination that a threshold number of write requests are obtained within the predetermined time window, and, in response to the first determination, sending an offload request to a virtual machine (VM), wherein the offload request specifies at least the write request.

Method and system for offloading parallel processing of multiple write requests

A method for processing data includes monitoring, by a virtual machine (VM), a plurality of computing resources, receiving an offload request by the VM, selecting, based on the monitoring, a computing resource from the plurality of computing resources, issuing, by the VM and in response to the offload request, the processing request to the computing resource, and servicing, by the computing resource, the processing request to obtain a result, wherein the VM and the computing resource are executing on a computing device.

Method and system for offloading parity processing

Disclosed are techniques for data deduplication, which include methods, systems, or computer products for reducing data redundancy in a data storage system comprising searching a cluster of nearest neighbors, wherein the cluster has been created using a locality sensitive hashing algorithm, to determine if a data block has been stored in the data storage system prior to writing the data block. In alternate embodiments, the nearest neighbor clusters could be created using one or more of the following algorithms: k-means clustering algorithm, a k-medoids clustering algorithm, a mean shift algorithm, a generalized method of moment (GMM) algorithm, or a density based spatial clustering of applications with noise (DBSCAN) algorithm.

Deduplication using nearest neighbor cluster

A fingerprint trie is used to store fingerprints for data portions stored on a storage system for use in implementing data deduplication on a storage system. The fingerprint trie may be used to compare fingerprint values to determine duplicate data portions, for example, in response to I/O operations. Leaf nodes of the fingerprint trie may be keyed by fingerprints, and a value of each leaf node may be a reference to the physical storage location of the data portion from which the fingerprint was generated. When an I/O operation is received, a fingerprint may be generated for each of one or more data portions included in the I/O operation. A fingerprint trie may be searched, for example by traversing multiple nodes of the trie according to pointers provided by the nodes, to determine whether there is any matching fingerprint specified in the fingerprint trie.

Deduplication using fingerprint tries

Allocation of storage array hardware resources between host-visible and host-hidden services is managed to ensure that sufficient hardware resources are allocated to host-visible services. Information obtained from monitoring real-world operation of the storage array is used to generate a model of the storage array. The generated model represents temporal dependencies between storage array hardware, host-visible services, and host-hidden services. Because the model includes information gathered over time and represents temporal dependencies, future occurrence of repeating variations of storage-related service usage and requirements can be predicted. The model may be used to generate hardware recommendations and dynamically re-allocate existing hardware resources to more reliably satisfy a predetermined level of measured performance.

Dynamic workload management based on predictive modeling and recommendation engine for storage systems

Described herein is a system that facilitates recommending data compression using machine learning and statistical attributes. According to an embodiment, a system can comprise receiving a dataset, statistical attributes associated with the dataset, and a compression requirement for compression of the dataset. The system can further comprise based on the statistical attributes and the compression requirement, estimating a first compression attribute and a second compression attribute of a group of compression processes. The system can further comprise selecting a primary compression process from the group of compression processes, based on an output of an analytics component, wherein the analytics component employs a neural network to determine the primary compression process based on analysis of the statistical attributes, the compression requirement, and a compression objective.

Recommending data compression scheme using machine learning and statistical attributes of the data

A method of workload aware dynamic CPU processor core allocation includes the steps of predicting estimated individual workloads for each emulation in a set of emulations for each decision period of a set of decision periods over a predictive time span. The method includes using, by a Mixed Integer Programming (MIP) engine, the predicted estimated individual workloads for each emulation in the set of emulations, a set of constraints, and an optimization function, to determine sets of CPU processor cores to be allocated to each emulation during each decision period over the predictive time span. The method further includes dynamically allocating, by the host computer system, the sets of CPU processor cores to each emulation during each decision period over the predictive time span based on the output from the MIP engine.

Growing community of inventors

Benares, India

Sweetesh Singh