Emc IP Holding Company LLC

Emc Corporation

Dell Products Inc.

Yubing Wang

Philippe Armangau

Jean-Pierre Bono

Christopher Alan Seibel

Ajay Karri

Vamsi K Vankamamidi

Sitaram D Pawar

Yining Si

William C Davenport

Sairam Veeraswamy

Michael P Wahl

Bruce E Caram

John F Gillono

Maher Kachmar

Socheavy D Heng

Sandeep Nirmale

Yaming Kuang

Soumyadeep Sen

Pranit Sethi

Kamakshi Viswanadha

Yunfei Chen

Chris Seibel

Christopher Jones

Kumar V K H Kanteti

John A Hassey

James McCoy

A method for assigning data to similarity groups (sim-groups). The method includes: partitioning a data stream, originating from a data source, into stream chunks; for at least a first stream chunk of the stream chunks: initializing a first chunk bin-map for the first stream chunk; partitioning the first stream chunk into first chunk segments; updating the first chunk bin-map based on a chunk segment processing of the first chunk segments; computing, to obtain a first similarity distance set, a similarity distance between the first chunk bin-map and each sim-group bin-map in a first set of sim-group bin-maps; identifying a first sim-group corresponding to a first similarity distance in the first similarity distance set; and assigning the first stream chunk to the first sim-group.

Data assignment to sim-groups using fingerprint distribution mapping

A method, computer program product, and computing system for allocating a first number of tokens from a plurality of tokens for processing read IO requests from a read IO queue, thus defining a number of allocated read tokens. A second number of tokens may be allocated from the plurality of tokens for processing write IO requests from a write IO queue, thus defining a number of allocated write tokens. It may be determined that the processing of the write IO requests is throttled. In response to determining that the processing of the write IO requests from the write IO queue is throttled, a maximum allowable number of write tokens may be defined. Additional tokens may be allocated for processing the read IO requests from the read IO queue based upon, at least in part, the maximum allowable number of write tokens and the number of allocated write tokens.

System and method for allocating storage system resources during write throttling

A technique for managing a metadata transaction log consolidates multiple mapping pointer changes that affect a single metadata block in a single transaction entry. The technique creates a data structure that identifies the mapping pointers in the metadata block that are changing, and stores the data structure in the transaction entry along with records that describe the individual mapping pointer changes.

Bulk updating of mapping pointers with metadata transaction log

A method is used in managing deduplication characteristics in a storage system. Deduplication entries stored in a deduplication cache are categorized into a set of deduplication groups based on a data deduplication probability associated with the deduplication entries. A machine learning system is used to dynamically adjust deduplication characteristics associated with the set of deduplication groups based on an I/O workload associated with the storage system.

Managing deduplication characteristics in a storage system

A method, computer program product, and computing system for defining a token pool size for a storage system token pool associated with a storage system. An input/output (IO) processing load for the storage system may be determined. The token pool size for the storage system token pool associated with the storage system may be dynamically adjusted based upon, at least in part, one or more changes in the IO processing load determined for the storage system.

System and method for dynamically adjusting priority-based allocation of storage system resources

Coordinating a reclaiming of data storage space among processing nodes of a data storage system includes, by a first node in response to an event, performing local cleanup of first-node mapping data and issuing a request message to a second node, the request message identifying the data storage space to be reclaimed. The event may be a notification from a RAID component rebuilding a RAID volume. In response to the request message, the second node performs local cleanup of second-node mapping data and issues a completion message to the first node indicating that the second node has completed its part of the reclaiming of the data storage space. The first node responds to the completion message by marking the data storage space as being reclaimed and signaling to a source of the event that the data storage space is available for new use.

Coordinated reclaiming of data storage space

A data storage system performs updating of a set of global counters stored in persistent storage accessible to a plurality of processing nodes of a data storage system, the global counters storing respective global count values describing associated units of data storage shared by the processing nodes for performing data storage operations. The updating includes, by each processing node, using a respective set of in-memory delta counters storing respective delta count values to track changes to respective global count values due to data storage operations performed by the processing node with respect to the units of data storage. A summarization process is regularly executed to merge the delta count values of the delta counters of all processing nodes into the respective global counters.

Managing global counters using local delta counters

Techniques for updating persistent statistics on a multi-transactional and multi-node storage system. The techniques can be practiced in a distributed storage system including a plurality of storage nodes. Each storage node can include a persistent storage configured to accommodate a set of delta counter pages, as well as a global counter page for summarizing delta count values tracked by respective delta counters. The techniques can include, in each storage node, tracking, by each of a set of delta counters, changes to delta count values due to storage node operations performed on units of data storage, and summarizing, periodically or at least at intervals, the delta count values of the respective delta counters as global count values. The techniques can further include summarizing the global count values across the respective storage nodes in a count summarization report and sending the count summarization report to a client via a communications interface.

Systems and methods of updating persistent statistics on a multi-transactional and multi-node storage system

Embodiments are directed to techniques for performing deduplication. A method includes (a) obtaining a digest of a data block logically-positioned within a filesystem, the digest providing a hash value of data of the data block, (b) searching a Most Wanted Digest Cache (MWDC) within system memory for the digest, (c) locating an entry in the MWDC using the digest, wherein this locating indicates that the data block has the same data as another data block located elsewhere within the filesystem, the other data block having been previously persistently-stored, the entry having been added to the MWDC in response to the other data block having been deduplicated at least a plurality number of times, (d) locating a mapping structure referenced by the entry located from the MWDC, the mapping structure providing metadata about the other data block, and (e) deduplicating the data block and the other data block with reference to the located mapping structure.

Speeding deduplication using a most wanted digest cache

A technique for managing a deduplication digest cache (DDC) includes assigning each digest entry of the DDC to one of multiple entry lists, scoring the entry lists based at least in part on the deduplicability of the data represented by the digest entries in the entry lists, and selecting for eviction from the DDC entries assigned to the lowest-scoring entry list. In this manner, entries assigned to entry lists that perform more poorly in terms of deduplicability tend to be evicted more quickly than entries assigned to entry lists that perform better.

Managing eviction from a deduplication cache

A technique for managing metadata in a data storage system includes receiving a set of data to be stored in a file system of the data storage system and calculating values of metadata blocks that the file system will use to organize the set of data in the file system. The technique aggregates values of the metadata blocks and sends them in a single, atomic transaction to a persistent cache. The transaction either succeeds, in which case the persistent cache stores the values of all of the metadata blocks, or it fails, in which case the persistent cache stores none of the values of the metadata blocks. Over time, the persistent cache flushes the values of the metadata blocks to a set of non-volatile storage devices that back the metadata blocks in the data storage system.

Managing file system metadata using persistent cache

Processing for a file system may include determining an inconsistency between a data log and inodes (index nodes) of the file system. Responsive to determining the inconsistency, recovery processing may be performed including first processing of the data log, second processing of the inodes of the file system, and third processing of the data log after said second processing. First processing may, for each entry of the data log that records a create snapshot operation to create a snapshot of a file having an associated inode of the file system that is a dirty snapshot inode, marking the associated inode as 'snap validated'. Second processing may include freeing each dirty snapshot inode not marked as 'snap validated'. Third processing may include freeing any entry of the data log where the entry references dirty snapshot inode freed by the second processing.

Recovery processing for persistent file data cache to reduce data loss

A technique for storing data selectively tags write requests received by a data storage system based on whether each write request is directed to a file having a file type on an exclusion list, thereby producing a set of tagged write requests directed to a file type on the exclusion list and a set of untagged write requests directed to a file type not on the exclusion list. The method further includes persistently storing tagged data specified by the set of tagged write requests without first compressing the tagged data, and persistently storing untagged data specified by the set of untagged write requests after compressing the untagged data.

Write tagging for selective avoidance of inline compression

A technique for performing overwrites on shared data blocks in a file system transfers reference weights of block pointers to other block pointers that share those data blocks. To overwrite a portion of a file stored in a shared data block pointed to by a first block pointer, a file system manager locates a second block pointer in the file system that points to the same data block. The first block pointer has a first reference weight, and the second block pointer has a second reference weight. The file system manager transfers the first reference weight to the second block pointer, e.g., by replacing the second reference weight with the sum of the first reference weight and the second reference weight.

Growing community of inventors

Southborough, MA, United States of America

Yubing Wang