Panzura, Inc.

Nortel Networks Corporation

Andrew P Davis

John Richard Taylor

Randy Yen-pang Chou

Brian Christopher Parkison

Richard James Sharpe

Allan Burrington

The disclosed embodiments disclose techniques for managing a global namespace for a distributed filesystem. Two or more cloud controllers collectively manage distributed filesystem data that is stored in a cloud storage system; the cloud controllers ensure data consistency for the stored data, and each cloud controller caches portions of the distributed filesystem. Furthermore, a global namespace for the distributed filesystem is also split across these cloud controllers, with each cloud controller 'owning' (e.g., managing write accesses for) a distinct portion of the global namespace and maintaining a set of namespace mappings that indicate which portion of the namespace is assigned to each cloud controller. During operation, an initial cloud controller receives a request from a client system to access a target file in the distributed system. This initial cloud controller uses the namespace mappings for the global namespace to determine a preferred cloud controller that will handle the request.

Managing a global namespace for a distributed filesystem

The disclosed embodiments disclose techniques for optimizing caching performance for a distributed filesystem. Two or more cloud controllers collectively manage distributed filesystem data that is stored in one or more cloud storage systems; the cloud controllers ensure data consistency for the stored data, and cloud controllers cache portions of the distributed filesystem on hard drives with rotating disk platters. The outside portions of these disk platters have higher I/O bandwidth than the inside portions of the disk platters. During operation, a cloud controller determines that a hot subset of cached data is likely to be accessed frequently by clients, and stores this hot subset to an outside portion of a disk platter. The cloud controller further determines that a cold subset of cache data is less likely to be accessed by clients, and stores the cold subset on an inside portion of a disk platter.

Short stroking and data tiering for a distributed filesystem

The disclosed embodiments disclose techniques for transferring and caching a cloud file in a cloud controller. Two or more cloud controllers collectively manage distributed filesystem data that is stored in one or more cloud storage systems; the cloud controllers cache and ensure data consistency for the stored data. During operation, a cloud controller receives a client request for a data block of a target file that is stored in the distributed filesystem but not currently cached in the cloud controller. The cloud controller initiates a request to a cloud storage system for a cloud file containing the requested data block. While receiving the cloud file from the cloud storage system, the cloud controller uses a set of block metadata in the portion of the cloud file that has already been received to determine the portions of the cloud file that should be downloaded to and cached in the cloud controller.

Transferring and caching a cloud file in a distributed filesystem

The disclosed embodiments provide a system that uses overlay metadata in a cloud controller to generate incremental snapshots for a distributed filesystem. Two or more cloud controllers collectively manage distributed filesystem data that is stored in one or more cloud storage systems. More specifically, the cloud controllers cache and ensure data consistency for the data stored in the cloud storage systems, with each cloud controller maintaining a metadata hierarchy that reflects the current state of the distributed filesystem. During operation, a cloud controller receiving new data from a client: (1) stores the new data in the cloud controller; (2) creates a metadata entry for the new data in the locally maintained metadata hierarchy; (3) updates the overlay metadata to point to the metadata entry and the new data stored in the cloud controller; and (4) then uses the overlay metadata to generate an incremental snapshot for the new data.

Using overlay metadata in a cloud controller to generate incremental snapshots for a distributed filesystem

The disclosed embodiments disclose techniques for executing a cloud command for a distributed filesystem. Two or more cloud controllers collectively manage distributed filesystem data that is stored in one or more cloud storage systems; the cloud controllers ensure data consistency for the stored data, and each cloud controller caches portions of the distributed filesystem. During operation, a cloud controller presents a distributed-filesystem-specific capability to a client system as a file in the distributed filesystem (e.g., using a file abstraction). Upon receiving a request from the client system to access and/or operate upon this file, the client controller executes an associated cloud command. More specifically, the cloud controller initiates a specially-defined operation that accesses additional functionality for the distributed filesystem that exceeds the scope of individual reads and writes to a typical data file.

Executing a cloud command for a distributed filesystem

The disclosed embodiments disclose techniques for restoring an archived file in a distributed filesystem. Two or more cloud controllers collectively manage distributed filesystem data that is stored in one or more cloud storage systems; the cloud controllers ensure data consistency for the stored data, and each cloud controller caches portions of the distributed filesystem. Furthermore, cloud controllers may archive infrequently-accessed files in an archival cloud storage system. During operation, a cloud controller receives a request from a client system to access an archived file, and restores this archived file from the archival cloud storage system.

Restoring an archived file in a distributed filesystem

The disclosed embodiments disclose techniques for synchronizing file updates between two cloud controllers of a distributed filesystem. Two or more cloud controllers collectively manage distributed filesystem data that is stored in the cloud storage systems; the cloud controllers ensure data consistency for the stored data, and each cloud controller caches portions of the distributed filesystem. During operation, a cloud controller receives a request from a client to access a file in the distributed filesystem. The cloud controller sends a synchronization update request for the file to a second cloud controller and in response receives a synchronization update for the file from the second cloud controller.

Synchronizing file updates between two cloud controllers of a distributed filesystem

The disclosed embodiments disclose techniques for managing metadata and data storage for a cloud controller in a distributed filesystem. Two or more cloud controllers collectively manage distributed filesystem data that is stored in one or more cloud storage systems. More specifically, the cloud controllers cache and ensure data consistency for the data stored in the cloud storage systems, with each cloud controller maintaining (e.g., storing) in a local storage device: (1) one or more metadata regions containing a metadata hierarchy that reflects the current state of the distributed filesystem; and (2) cached data for the distributed filesystem. During operation, the cloud controller receives an incremental metadata snapshot that references new data written to the distributed filesystem. The cloud controller stores updated metadata from this incremental metadata snapshot in one of the metadata regions on the local storage device.

Managing metadata and data storage for a cloud controller in a distributed filesystem

The disclosed embodiments provide a system that adjusts the characteristics of a distributed filesystem using a locality policy. Two or more cloud controllers collectively manage distributed filesystem data that is stored in one or more cloud storage systems; the cloud controllers cache and ensure data consistency for the stored data. During operation, a cloud controller receives a locality policy that specifies one or more management policies for the cloud controller. The portion of the distributed filesystem's data that is managed, created, and/or cached at the cloud controller is then managed based on this locality policy. Locality policies facilitate customizing and optimizing data management for the distributed filesystem to fit the needs of an organization (e.g., specific sets of users, applications, and/or datasets).

Customizing data management for a distributed filesystem

The disclosed embodiments disclose techniques for deleting a file from a distributed filesystem. Two or more cloud controllers collectively manage distributed filesystem data that is stored in one or more cloud storage systems; the cloud controllers store metadata for the distributed filesystem, and cache and ensure data consistency for the data stored in the cloud storage systems. During operation, a cloud controller receives a request from a client to delete a file from the distributed filesystem. The cloud controller updates a user view of the distributed filesystem to present the appearance of the target file being deleted to the client, and then initiates a background deletion operation to delete the target file without negatively affecting the performance of the other users of the distributed filesystem.

Deleting a file from a distributed filesystem

The disclosed embodiments provide techniques for performing deduplication for a distributed filesystem. Two or more cloud controllers collectively manage distributed filesystem data that is stored in one or more cloud storage systems; the cloud controllers cache and ensure data consistency for the stored data. During operation, a cloud controller receives an incremental metadata snapshot that references new data that was added to the distributed filesystem by a remote cloud controller. The cloud controller extracts a set of deduplication information from this incremental metadata snapshot. Upon receiving a subsequent client write request (e.g., a file write that includes one or more data blocks), the cloud controller uses the extracted deduplication information to determine that one or more data blocks in the client write request have already been written to the distributed filesystem.

Performing deduplication in a distributed filesystem

The disclosed embodiments disclose techniques that facilitate the recovery of a virtual machine using a distributed filesystem. Two or more cloud controllers collectively manage distributed filesystem data that is stored in one or more cloud storage systems; the cloud controllers ensure data consistency for the stored data, and each cloud controller caches portions of the distributed filesystem in a local storage pool. During operation, a host server executes program instructions for an application in a virtual machine (VM); data associated with this application and/or this virtual machine is stored in the distributed filesystem. Upon detecting a subsequent failure, the system can recover and resume the execution of the virtual machine and application using the previous application and virtual machine data that was stored in the distributed filesystem.

Facilitating the recovery of a virtual machine using a distributed filesystem

The disclosed embodiments provide a system that archives data for a distributed filesystem. Two or more cloud controllers collectively manage distributed filesystem data that is stored in one or more cloud storage systems; the cloud controllers cache and ensure data consistency for the stored data. During operation, a cloud controller receives a request from a client for a data block of a file stored in the distributed filesystem. Upon determining that the requested data block is not currently cached in the cloud controller, the cloud controller sends a peer cache request for the requested data block to a peer cloud controller in the distributed filesystem.

Accessing cached data from a peer cloud controller in a distributed filesystem

The disclosed embodiments provide a system that distributes data for a distributed filesystem across multiple cloud storage systems. Two or more cloud controllers collectively manage distributed filesystem data that is stored in one or more cloud storage systems; the cloud controllers cache and ensure data consistency for the stored data. Whenever each cloud controller receives new data from a client, it outputs an incremental metadata snapshot for the new data that is propagated to the other cloud controllers and an incremental data snapshot containing the new data that is sent to a cloud storage system. During operation, a backup cloud controller associated with the distributed filesystem is also configured to receive each (incremental) metadata snapshot, such that, upon determining the failure of a cloud controller, the backup cloud controller can immediately begin receiving data requests from clients associated with the failed cloud controller.

Growing community of inventors

Santa Cruz, CA, United States of America

Andrew P Davis