Emc IP Holding Company LLC

Sun Microsystems, Inc.

Oracle America, Inc.

Dell Products Inc.

Anton Rang

Andrew B Hastings

Harriet G Coverston

Jonathan Mark Walton

Dan Sledz

Daniel Powell

Max Laier

David Vengerov

Radia Joy Perlman

Alok Choudhary

Mayank Rajawat

Kevin J Clarke

Steve Mcpolin

Robert Gittins

Ryan Libby

Lachlan McIlroy

Chris Stacey

Attilio Rao

Dhanashri Parasharam Patil

Simarjot Kaur

Brian D Reitz

The described technology is generally directed towards maintaining a heat score per file based on a file's access/activity level, and using the heat score to re-tier files from faster to slower storage and vice-versa. Files on a slower tier that have more activity are thereby moved ('up-tiered') to a faster access storage tier, while the files that have less activity on the faster access storage tier are moved (“down-tiered”) to a slower access storage tier. The heat score for a file is increased based on more input-output operation activity, and decayed over time. The decaying can be based on a relative importance value (e.g., business value) assigned to the file or a domain that includes the file. Ordering the files based on their respective heat scores allows only a fraction of the files to be considered for re-tiering before the re-tiering moves are performed.

Heat score-based tiering of data between different storage tiers of a file system

Facilitating per-CPU reference counting for multi-core systems with a long-lived reference is provided herein. A system includes a processor and a memory that stores executable instructions that, when executed by the processor, facilitate performance of operations. The operations include determining a first quantity of releases associated with an object in a data structure of the system and determining a second quantity of acquisitions associated with the object. The first quantity of releases can be distributed among respective first counters of processing elements of a group of processing elements. The second quantity of acquisitions can be distributed among respective second counters of the processing elements of the group of processing elements. Further, the operations can include, based on the second quantity of acquisitions and the first quantity of releases being determined to be a same value, implementing a removal of the object from the data structure.

Facilitating per-CPU reference counting for multi-core systems with a long-lived reference

A storage system calculates relative writability of SSDs and biases storage of data from write IOs to the SSD that has the greatest relative writability, where writability is a value calculated as a function of remaining wear-life and drive capacity. When the remaining wear-life of an SSD falls below a threshold, unstable data is evicted from that drive, where data stability is an indication of likelihood of data being changed. The drive with the greatest relative writability is selected as the target for the unstable data. The drive with the greatest relative writability is also selected as the donor for stable data that is moved to the free space created by eviction of the unstable data. Consequently, the SSD that triggers the low wear-life threshold processes fewer write IOs.

Wear-balancing across storage devices based on endurance measurement

Pre-decompressing a compressed form of data that has been pre-fetched into a cache to facilitate subsequent retrieval of a decompressed form of the data from the cache is presented herein. A system retrieves, from a first portion of a cache, a compression chunk comprising compressed data blocks representing a compressed form of a group of data blocks in response to a first cache hit from the first portion of the cache being incurred, decompresses the compression chunk to obtain a decompressed chunk comprising uncompressed data blocks representing an uncompressed form of the group of data blocks, and inserts the uncompressed data blocks into a second portion of the cache. Further, the system retrieves, from the second portion of the cache, an uncompressed data block of the uncompressed data blocks in response to a second cache hit from the second portion of the cache being incurred.

Pre-decompressing a compressed form of data that has been pre-fetched into a cache to facilitate subsequent retrieval of a decompressed form of the data from the cache

Compression of data for a file system utilizing protection groups can be implemented and managed. A compression management component (CMC) can control compression of data via inline or post-process compression for storage in protection groups in memory, including determining whether to compress data, determining a compression algorithm to utilize to compress data, and/or determining whether to perform inline and/or post-process compression of data. CMC can generate protection group (PG) metadata for a PG in which compressed data is stored. PG metadata can comprise a logical extent map that describes which logical blocks contain compressed data, a list of cyclic redundancy check values for logical blocks, and a list of compression chunks that store individual metadata regarding individual compressed streams, wherein, for an individual compressed stream, the individual metadata comprises a compression format, compressed size, uncompressed size, and/or starting offset in physical space within the PG.

Compression of data for a file system

Implementations are provided herein for offering partitioned performance within a distributed file system and more specifically, for offering adaptive predicted impact of resource consumption by pending work items. Core resource consumption per work item can be estimated prior to processing the work item. When processing the work item, the actual amount of resources used to process the work item can be measured and recorded. The file system can then update future estimates for performing work items based on past results. Resources made available to process future requests can be throttled based on dynamically updated estimates of resource consumption by pending work items.

Partitioned performance: adaptive predicted impact

Implementations are provided herein for offering partitioned performance within a distributed file system and providing throttling at the granular level. A set of hardware and network resources available to process work items can be determined. A set of resource accounting tokens based on resource records generated when processing work items can be dynamically updated. A granular resource accounting aggregate for a customizable field of data can be selected for throttling, such as a unique user identifier, a unique group identifier, a unique client internet protocol address, a unique file, etc. A granular throttling level can then be established based on a granular throttling policy. In response to the resource accounting aggregate meeting the throttling level, the user, group, internet protocol address, etc. can be throttled at, at least one of, the cluster layer, the node layer or the protocol layer.

Partitioned performance: using resource account aggregates to throttle at the granular level

Implementations are provided herein for offering partitioned performance within a distributed file system. Core resource consumption per work item can be tracked independently. Discriminative data already known by the file system surrounding the context of the work item can be used to determine a reference resource accounting specification applicable to the work item. When processing the work item, a detailed resource record can be generated that inventories the resources used in processing the work item. The resource record associated with the work item can be recorded into a set of resource accounting tokens that track activity resource consumption at a granular level. A universal table of resource accounting tokens can be dynamically updated upon the processing of work items and generation of associated resource records throughout the distributed file system.

Partitioned performance tracking core resource consumption independently

Implementations are provided herein for offering partitioned performance within a distributed file system; specifically, for providing adaptive policies and leases within the partitions. An amount of resources available to a cluster of nodes operating as a distributed file system can be determined and those resources can be apportioned to individual nodes based on hardware profiles of the nodes. A set of resource accounting tokens can be dynamically updated and used as a basis to generate a cluster resource accounting aggregate, a set of node resource accounting aggregates, and a set of protocol resource accounting aggregates. The dynamically updated resource accounting aggregates can then be used to dynamically throttle resource available to process work requests at the cluster, node, and protocol head layers based on policy.

Partitioned performance adaptive policies and leases

A method for archiving a file that has multiple file parts stored on multiple object storage devices of a computer system includes the steps of saving the file parts on one or more archive devices, allowing one or more changes to be made to the file during the saving step, and applying at least one of the changes made during the saving step to one of the file parts stored on a respective object storage device. Furthermore, the method is performed such that the file parts saved on the one or more archive devices do not include the one or more changes made to the file during the saving step.

File archiving system and method

Various systems and methods of a distributed file system architecture are disclosed. A method for creating a file may include receiving a message to open or create a file with a given name from a file system call handler. Once the message has been received a determination may then be made as to whether a file with the given name exists. If no file with the given name exists, then a new file with the given name may be created and assigned a globally unique file number.

Delegation in a file system with distributed components

A distributed file system is disclosed which may include one or more input/output (I/O) nodes and one or more compute nodes. The I/O nodes and the compute nodes may be communicably coupled through an interconnect. Each compute node may include applications to perform specific functions and perform I/O functions through libraries and file system call handlers. The file system call handlers may be capable of providing application programming interfaces (APIs) to facilitate communication between the plurality of I/O nodes and the applications. The file system call handlers may use a message port system to communicate with other compute nodes.

File system with distributed components

One embodiment of the present invention provides a system that facilitates securely forgetting a secret. During operation, the system obtains a set of secrets which are encrypted with a secret key S, wherein the set of secrets includes a secret to be forgotten and other secrets which are to be remembered. Next, the system decrypts the secrets to be remembered using S, and also removes the secret to be forgotten from the set of secrets. The system then obtains a new secret key S, and encrypts the secrets to be remembered using S. Finally, the system forgets S.

Method and apparatus for securely forgetting secrets

A method for processing using a shared file that includes allocating a first working buffer between the shared file and a plurality of address spaces, wherein each of the plurality of address spaces is associated with one of a plurality of processors, copying first data from the shared file to the first working buffer by a first aggregator copying the first data from the first working buffer to the plurality of address spaces by the first aggregator, processing the first data, in parallel, by the plurality of processors to obtain a result, wherein the plurality of processors access data from the plurality of address spaces, and storing the result in the shared memory.

Method and system for file access using a shared memory

A method for routing an incoming service request is described wherein the service request is routed to a selected storage tier based on that selected storage tier having a predicted value indicating a state having greater utility as compared with the predicted value of the state associated with at least one other storage tier within the storage system. A computer system comprising a multi-tier storage system is described, the multi-tier storage system having a routing algorithm configured to adaptively tune functions which map variables describing the state of each storage tier of the storage system into the average latency experienced by incoming service requests associated with the storage tier.

Growing community of inventors

Houlton, WI, United States of America

Anton Rang