Emc IP Holding Company LLC

Emc Corporation

Jonathan Mark Walton

Anton Rang

Dan Sledz

Daniel Powell

Lijun Wang

Edward Cande

Aaron Passey

Neal Thomas Fachan

Tyler Arthur Akidau

Eric Michael Lemar

Justin Husted

Jeffrey Douglas Hughes

Max Laier

Ronald Steinke

Evgeny Popovich

Alain Hamel

Suraj Raju

Austin Voecks

Michael Frank

Nathan Edward Dire

Oliver Chung

Steven Hubbell

Tanuj Khurana

Hualiang Xu

Raeanne Marks

Yi Li

Karthik Palaiappan

Cornelis Van Rij

Direct read in clustered file systems is described herein. A method as described herein can include determining, for a write operation on a resource stored by a data storage system, as initiated by an initiator node, a reference count for the resource, the reference count comprising a number of target storage regions of the data storage system to be modified by write data during the write operation; facilitating conveying, from the initiator node to a lock coordinator node, the reference count for the resource; facilitating conveying, from the initiator node to respective participant nodes that are respectively assigned to the target storage regions, the write data and a key value for the write operation; and facilitating causing the respective participant nodes to convey respective notifications that comprise the key value in response to the respective participant nodes writing the write data to the target storage regions.

Direct read in clustered file systems

Incremental replication between foreign system dataset stores is described herein. A method as described herein can include connecting to a target storage system to facilitate an initial transfer of a dataset from the system to the target storage system, wherein the system is distinct from the target storage system; in response to determining that a first version of the dataset is present on the target storage system, comparing the first version of the dataset to a second version of the dataset stored on the system; and, in response to determining that the second version of the dataset is more recent than the first version of the dataset, transferring a differential between the first version of the dataset and the second version of the dataset to the target storage system, the differential comprising less than all of the second version of the dataset.

Incremental replication between foreign system dataset stores

Implementations are provided herein for using inode revision numbers associated with a modified LIN and a set of Parent LINs to causally order transactions within a distributed file system. Any time an inode is changed, its inode revision number can be incremented by 1. When events within file system are processed causing an inode or a set of inodes to be modified, an event transaction log entry can made. The event transaction log entry can denote a description of the event, a set of modified inode and inode revision number pairs, and a set of parent inode and inode revision number pairs. Entries in the event transaction log can be used to build an inode map for each inode implicated in the event transaction log. The inode map can be used to build a set of direct causal dependencies for each transaction in the event transaction log. The set of direct causal dependencies can be used to generate a causal ordering of the transactions in the event transaction log, that in some implementations, can be made available to external services.

Causally ordering distributed file system events

Implementations are provided herein for offering partitioned performance within a distributed file system and more specifically, for offering adaptive predicted impact of resource consumption by pending work items. Core resource consumption per work item can be estimated prior to processing the work item. When processing the work item, the actual amount of resources used to process the work item can be measured and recorded. The file system can then update future estimates for performing work items based on past results. Resources made available to process future requests can be throttled based on dynamically updated estimates of resource consumption by pending work items.

Partitioned performance: adaptive predicted impact

Implementations are provided herein for offering partitioned performance within a distributed file system and providing throttling at the granular level. A set of hardware and network resources available to process work items can be determined. A set of resource accounting tokens based on resource records generated when processing work items can be dynamically updated. A granular resource accounting aggregate for a customizable field of data can be selected for throttling, such as a unique user identifier, a unique group identifier, a unique client internet protocol address, a unique file, etc. A granular throttling level can then be established based on a granular throttling policy. In response to the resource accounting aggregate meeting the throttling level, the user, group, internet protocol address, etc. can be throttled at, at least one of, the cluster layer, the node layer or the protocol layer.

Partitioned performance: using resource account aggregates to throttle at the granular level

Implementations are provided herein for offering partitioned performance within a distributed file system. Core resource consumption per work item can be tracked independently. Discriminative data already known by the file system surrounding the context of the work item can be used to determine a reference resource accounting specification applicable to the work item. When processing the work item, a detailed resource record can be generated that inventories the resources used in processing the work item. The resource record associated with the work item can be recorded into a set of resource accounting tokens that track activity resource consumption at a granular level. A universal table of resource accounting tokens can be dynamically updated upon the processing of work items and generation of associated resource records throughout the distributed file system.

Partitioned performance tracking core resource consumption independently

Implementations are provided herein for offering partitioned performance within a distributed file system; specifically, for providing adaptive policies and leases within the partitions. An amount of resources available to a cluster of nodes operating as a distributed file system can be determined and those resources can be apportioned to individual nodes based on hardware profiles of the nodes. A set of resource accounting tokens can be dynamically updated and used as a basis to generate a cluster resource accounting aggregate, a set of node resource accounting aggregates, and a set of protocol resource accounting aggregates. The dynamically updated resource accounting aggregates can then be used to dynamically throttle resource available to process work requests at the cluster, node, and protocol head layers based on policy.

Partitioned performance adaptive policies and leases

Systems and methods are provided herein for efficient local caching of data tiered to cloud storage to help reduce the bandwidth cost of repeated reads and writes to the same region of a stubbed file, increase the performance of write operations, and increase performance of read operations to portions of a stubbed file accessed repeatedly. When operations are directed toward data tiered to the cloud, the data can be read from cloud storage and stored within a local cache. A cache tracking tree can be generated and used to track file regions of a stub file, cached states associated with regions of the stub file, a set of cache flags, and other file and mapping data. For example, the cache state of regions of a stub file can be tracked including a cached data state, a non-cached state, a modified state, or a truncated state. Operations directed toward stubbed files can then look to the cache tracking tree to determine the most efficient way to access, retrieve, or operate on the data that maximizes local file system performance while reducing network activity.

Using a local cache to store, access and modify files tiered to cloud storage

Systems and methods are provided herein for establishing data constructs necessary for efficient and transparent tiering of data from local storage to cloud storage, and there use in transferring data to and from cloud storage. Files data can split into chunks and stored as individual objects in cloud storage. A stub file can replace the local file in local storage and describe a mapping of cloud objects that contain data to specific regions of the file. Upon creation of the stub and transfer of data to cloud storage, local storage capacity previously dedicated to file data can be freed for other uses.

Data and metadata structures for use in tiering data to cloud storage

Embodiments are directed towards managing designations of a plurality of files across multiple clusters. Files may be designated as primary files or secondary files. Primary files at one cluster may be asynchronously replicated at another cluster as secondary files. Secondary files may be synchronously validated with remote primary files based on local file access requests. If a write request for a local secondary file is provided, then the local file may be designated as the primary file and the remote file may be designated as the secondary file. Local access may then be provided to the local primary file. If a read request for a local secondary file is provide, then local access may be provided to the local secondary file without changing file designations. In some embodiments, file designations may be preemptively changed based on a file change designation schedule for clusters at different geographic locations.

Geographic distribution of files

In general, embodiments of the invention relate to reading data from and writing data to a storage system. Specifically, embodiments of the invention relate to a read only mode for a portion of a storage system. In one embodiment, a selective read-only mode for a portion of a storage system is implemented by monitoring a condition that affects a subset of persistent storage in a storage system, by detecting the condition, by entering a read-only mode for the subset, and by enforcing a policy of processing write requests and read requests to the storage system, which includes processing the write requests without modifying user data stored on the subset and processing the read requests, including requests for user data stored on the subset.

Growing community of inventors

Seattle, WA, United States of America

Jonathan Mark Walton