Emc IP Holding Company LLC

Emc Corporation

Edward Cande

Lijun Wang

Jonathan Mark Walton

Hualiang Xu

Max Laier

Evgeny Popovich

John Whitehead

Bharat Sharma

Yi Li

Implementations are provided herein for having at least two data streams associated with each file in a file system. The first, a cache overlay layer, can store additional state information on a per block basis that details whether each individual block of file data within the cache overlay layer is clean, dirty, or indicates that a write back to the storage layer is in progress. The second, a storage layer, can be a use case defined repository that can transform data using data augmentation methods as a part of tiering data to external repositories. File system operations directed to the cache overlay layer can be processed asynchronously from file system operations directed to the storage layer.

Persistent cache layer to tier data to cloud storage

Systems and methods are provided herein for efficient local caching of data tiered to cloud storage to help reduce the bandwidth cost of repeated reads and writes to the same region of a stubbed file, increase the performance of write operations, and increase performance of read operations to portions of a stubbed file accessed repeatedly. When operations are directed toward data tiered to the cloud, the data can be read from cloud storage and stored within a local cache. A cache tracking tree can be generated and used to track file regions of a stub file, cached states associated with regions of the stub file, a set of cache flags, and other file and mapping data. For example, the cache state of regions of a stub file can be tracked including a cached data state, a non-cached state, a modified state, or a truncated state. Operations directed toward stubbed files can then look to the cache tracking tree to determine the most efficient way to access, retrieve, or operate on the data that maximizes local file system performance while reducing network activity.

Using a local cache to store, access and modify files tiered to cloud storage

Systems and methods are provided herein for establishing data constructs necessary for efficient and transparent tiering of data from local storage to cloud storage, and there use in transferring data to and from cloud storage. Files data can split into chunks and stored as individual objects in cloud storage. A stub file can replace the local file in local storage and describe a mapping of cloud objects that contain data to specific regions of the file. Upon creation of the stub and transfer of data to cloud storage, local storage capacity previously dedicated to file data can be freed for other uses.

Data and metadata structures for use in tiering data to cloud storage

Systems and methods are provided herein for writing back data to files tiered in cloud storage. Files tiered to cloud storage can be stored in chunks where each chunk is stored as a distinct cloud object and where each cloud object is mapped to known offset regions of a file and the mapping is stored in local storage within a stub file. If the file is modified locally, e.g., a write has occurred, the offset range associated with the write can be used to determine which cloud object associated with the file is being modified. A new cloud object can be stored with the write data, while the previous cloud object is retained in cloud storage. The mapping, stored within the stub file can updated to reference the new cloud object for the region of the file that was modified. In addition, mapping can be retained for the previous version of the file allowing for efficient recall of past versions of files.

Growing community of inventors

Bedford, NH, United States of America

Edward Cande