Emc IP Holding Company LLC

Emc Corporation

Wisconsin Alumni Research Foundation

Abhishek Rajimwale

George Mathew

Murthy V Mamidi

Windsor Hsu

Donna Barry Lewis

Charles J Hickey

Xiaohong W Wang

Yamini Allu

Michael Mansfield Swift

Pradeep Thomas

Balaji Subramanian

Uday Jonnala

Sai Chivukula

Andrea C Arpaci-Dusseau

Remzi H Arpaci-Dusseau

Geetha Shendge

Vishal Chandra Tiwary

Xiaobing Zhang

Swaminathan Sundararaman

Sriram Subramanian

Described is a system (and method) that maintains deduplication efficiency when storing data within a clustered data storage environment that implements a global namespace. To provide such a capability, the system may obtain granular data source identifying information from a client system that provides data to be backed-up by a backup component. The data source identifying information may take the form of a placement tag that is associated with the received data. The backup component may then provide such placement tags when providing the backup data to the clustered storage system. The placement tags may then be used to intelligently distribute backup files to particular storage nodes of the clustered storage system to improve deduplication efficiency.

Granular data source identification for obtaining deduplication storage efficiency within a clustered environment

Different logical partitions representing parts of a distributed file system global namespace are hosted on some cluster nodes, e.g., metadata nodes. File content and shadow logical partitions corresponding to the different logical partitions are hosted on other nodes, e.g., data nodes. Each file is associated with a metadata and data node. TCP links are established between nodes. Upon opening files, a file manager server session is generated between each pair of nodes associated with the open files to track open states and is recorded in a mapping table. The mapping table identifies each open file and associated nodes. When a metadata or data node of a particular pair of nodes associated with an open file becomes unavailable, the mapping table is consulted to identify another of the metadata or data node associated with the open file. Crash recovery protocols are performed on the other of the metadata or data node.

Reliably maintaining strict consistency in cluster wide state of opened files in a distributed file system cluster exposing a global namespace

A method of capturing a fractional snapshot of a distributed file system includes receiving a snapshot request to generate a factional snapshot of a file system. The request includes an indication of a subset of data of the file system. The method further includes identifying, by a processing device, the subset of data of the file system indicated by the snapshot request and generating the fractional snapshot comprising the subset of data of the file system indicated by the snapshot request.

Fractional consistent global snapshots of a distributed namespace

An intelligent method of handling incremental backups concurrent with load balancing movement. The file system uses placement tags, incremental backup requests and capacity balancing data movement to make intelligent decision to avoid affecting any backup windows for clients or backup apps. The file system tracks capacity balancing file movements inside the cluster. When switching locations of files in a cluster from one node to another, it is performed as an atomic change of switching inode attributes by the file system after the contents of the file have been copied over to the new node. During the file movement for capacity balancing, the file system handles requests for full backups differently than requests for incremental backups. The file system continues to handle virtual systhesis and fastcopy requests on the node that hosts the previous backup to ensure that the incremental backup succeeds with the expected smaller backup window from the client.

Optimizing incremental backup for clients in a dedupe cluster to provide faster backup windows with high dedupe and minimal overhead

Described is a system (and method) that intelligently distributes data within a clustered storage environment. To provide such a capability, the system may distribute backup files by considering a source of the data to be backed-up. In particular, the system may leverage the ability of front-end components such as a backup application to perform a granular data source identification of data. Such information may be propagated to back-end components such as a storage filesystem in the form of a data source identifier (e.g. placement tag). The data source identifiers may then be accessed by the clustered storage system to intelligently distribute backup files amongst a set of storage nodes forming a cluster. For example, backup files from the same data source may be stored on the same storage node to obtain the same deduplication efficiency as a single storage system.

Utilizing data source identifiers to obtain deduplication efficiency within a clustered storage environment

Embodiments for enabling snapshot backups in a global name space of a cluster network, by representing the name space of cluster network in an MTree, storing data files organized in a B+ Tree format on one or more data nodes, storing name specific information of the data files in a B+ Tree format in a meta node, wherein a B+ Tree of the meta node accesses each corresponding B+ Tree in each of the one or more data nodes. The process takes snapshot backups of individual MTree limbs, and links the limbs of each snapshot into groups based on a cluster identifier and snapshot identifier.

Global snapshot backups of a distributed name space

Described is a system that provides the ability to distribute backup data according to performance characteristics associated with nodes that form a clustered storage environment. For example, the clustered storage environment may organize nodes into tiers that have certain storage performance-related characteristics such as read/write latency, deduplication processing efficiency, bandwidth, etc. Accordingly, the system may associate the backup data with an identifier for an appropriate storage performance tier. The identifier may then be utilized by the clustered storage environment to redirect storage operations to a node within the appropriate storage performance tier. Thus, the system allows for the distribution of backup data to particular storage performance tiers based on particular storage or workload requirements.

Storage performance-based distribution of deduplicated data to nodes within a clustered storage environment

Described is a system for distributing multiple phases of a deduplication processing amongst of set of nodes. The system may perform a load-balancing in configurations where multiple generations of backup data are redirected to the same host node, and thus, require the host node to perform certain storage processes such as writing new backup data to its associated physical storage. Accordingly, the system may perform an initial (or first phase) processing on a first node that is selected based on resource usage or classification (e.g. metadata storing node). The system may then perform a subsequent (or second phase) processing on a second, or host node, that is selected based on the node already storing previous generations of the backup data. Accordingly, the system still redirects processing to a host node, but provides the ability to delegate certain deduplication operations to additional nodes.

Distributing multiple phases of deduplication processing amongst a set of nodes within a clustered storage environment

Described is a system for reclaiming unused storage space when distributing backup data to storage performance-based tiers within a clustered storage environment. The clustered storage environment may organize nodes into storage performance tiers, and each node within a tier may be tuned for a particular type of workload. Accordingly, the system implements a file system that provides the ability to reclaim unused storage space (e.g. perform garbage collection) for specific nodes or tiers independently despite implementing a global namespace. The global namespace may group (or co-locate) aspects of file information with the backup files. For example, file information may be stored on the same node that stores the associated backup files. Accordingly, the system may reclaim unused storage space for each node, or set of nodes forming a storage performance tier, independently and efficiently.

Efficient name space organization in a global name space cluster

The distribution of global namespace to achieve performance and capacity linear scaling in cluster filesystems. Specifically, the disclosure entails the distribution of files, tracked in namespace trees, among various cluster nodes while linking the namespace trees through a centralized access point, thus providing a global namespace abstraction for the distributed cluster.

Distribution of global namespace to achieve performance and capacity linear scaling in cluster filesystems

According to some embodiment, a backup storage system receives, by a first phase of a data deduplication pipeline, a request from a client for reading or writing a data segment associated with a data stream stored in or to a storage system. In response to the request, the system retrieves, by the first phase, load parameters associated with a second phase in the data deduplication pipeline. For each of the load parameters associated with the second phase, the system determines, by the first phase, whether the load parameter has exceeded a load threshold associated with the second phase. The system throttles, by the first phase, performance of a specific job in the data deduplication pipeline by the second phase in response to a determination that at least one of the load parameters associated with the second job phase has exceeded the load threshold associated with the second phase.

Method and system for data deduplication based on load information associated with different phases in a data deduplication pipeline

According to one embodiment, recovering file systems without a replica is described. In one example, a method includes discovering an error in a first storage partition of the file system containing a current version of a namespace metadata, finding a second storage partition containing a most recent copy of the namespace metadata, and restoring the found copy of the namespace metadata from the second storage partition to the first storage partition for use as a current version of the namespace metadata.

System and method for recovering file systems without a replica

According to one embodiment, recovering file systems by restoring partitions is described. In one example, a method includes discovering an error in a storage partition of a file storage system, the storage partition including identification metadata to identify itself to the file system, finding a backup storage partition containing data and identification metadata corresponding to that of the error storage partition, restoring the error storage partition using the backup storage partition, and using the restored identification metadata to identify the restored storage partition to the file system.

System and method for recovering file systems by restoring partitions

According to one embodiment, a large scale data storage system with fault tolerance is described. In one example, a system includes a plurality of storage partitions, each partition including a portion of a large scale data store, and metadata corresponding to the portion of the large scale data store. At least one partition includes namespace metadata for the large scale data store including namespace data for a portion of the large scale data store outside the at least one partition.

Large scale data storage system with fault tolerance

In general, techniques are described for enabling a restartable file system. A computing device comprising a processor that executes an operating system may implement the techniques. The processor executes kernel and file system functions of the operating system to perform an operation, where both types of functions call each other to perform the operation. The operating system stores data identifying those of the kernel functions that called the file system functions. In response to determining that one of the file system functions that was called has failed, the operating system accesses the data to identify one of the kernel functions that most recently called one of the file system functions, and returns control to the identified one of the kernel functions without executing any of the file system functions called after the identified one of the kernel functions and prior to the one of the file system functions that failed.

Growing community of inventors

San Jose, CA, United States of America

Abhishek Rajimwale