Emc IP Holding Company LLC

Dell Products Inc.

Charles J Hickey

Murthy V Mamidi

George Mathew

Murthy Mamidi

Shashikanth Lakshmikantha

Krishna Chaitanya Gudipati

Abhishek Rajimwale

Anil Ravindranath

Neerajkumar Nareshkumar Chourasia

Mark Chamness

Orit Levin-Michael

Geetha Shendge

Dhawal Bhagwat

Xiaohong W Wang

Alice Cuicui Zhao

Sai Chivukula

Sampath Jayaram

Jose Medina

Ramesh Kannan K

Manivannan Chithirasen

Sirisha Kapoor

Page keys are formed in a key-value store (KVS) storing metadata of a filesystem. Page keys identify pages of trees within which namespaces of logical partitions are held. Leaf pages include first and second full keys for each file. First full keys index inodes. Second full keys index filenames. Inode keys are formed in the KVS. Inode keys identify inodes of the files. Metadata is embedded into the page and inode keys such that each page key includes an identifier of a logical partition, an identifier of a snapshot taken of a namespace of the logical partition, and a page number to a page tree holding the namespace. Each inode key includes the logical partition identifier, the snapshot identifier, and an inode number of an inode to a file. Upon damage to the namespace, queries are conducted of the KVS and the namespace is recovered based on the query results.

Namespace recovery for key-value store (KVS)-persisted metadata

Embodiments for providing adaptive namespace verification with high efficiency. The verification process is used in modern filesystems to check the integrity of key data structures, such for backup systems where providing thorough and efficient mechanisms for verification is critical. Embodiments accelerate the verification process by selectively applying continuous, differential, and integral verification techniques for startup and runtime repair purposes to overcome present problems for system capacities that exceed the petabyte range and impose significant time requirements using existing verification methods. Embodiments work with both single-node and distributed filesystems.

Continuous namespace verification for single-node filesystems

Recovering a namespace held in a tree structure that has been corrupted includes scanning a linear representation of pages belonging to the tree structure to copy out keys from uncorrupted leaf pages into a new tree structure. The scan is repeated over snapshots that have been taken of the tree to find keys that are missing from the new tree structure. Upon finding a missing key in an uncorrupted leaf page of a snapshot, the missing key is copied out into the new tree structure unless the missing key has already been found and copied out from an uncorrupted leaf page of a more recent snapshot taken of the tree structure.

Namespace salvage in logging-based filesystems

Continuous namespace verification for cluster filesystems

Adaptive namespace verification for single-node filesystems

Adaptive namespace verification for cluster filesystems

A method of generating a hybrid snapshot includes receiving a request to generate a snapshot of a distributed file system and identifying a first storage resource of the distributed file system and a second storage resource of the distributed file system based on the request. The method further includes generating the snapshot of the distributed file system, the snapshot including a data-full snapshot of the first storage resource and a data-less snapshot of the second storage resource.

Hybrid snapshot of a global namespace

Embodiments for enabling granular migration of data with high efficiency. A defined metadata element, a tag, is assigned to each file, and then tag filtering is used to direct the data to the proper location. Files with different tags can be selected for transfer, and such a group of tags is referred to as a tag set. Embodiments can be used with a defined backup system file migration process, such as present in the Data Domain File System. By using snapshots, incoming new data (ingested file) is allowed to continue while the migration is in process and maintaining data consistency at the same time. This is achieved by performing operations on B+ Tree snapshots in conjunction with tag filtering on keys present in the leaf pages of these structures. This method is efficient became it makes a single pass walk of a B+ Tree in contrast with previous methods that look up files one-by-one via their pathname.

Granular data migration

Embodiments for granular replication of data with high efficiency. A defined metadata element embodied as a tag is assigned to each file. Tag filtering is used to direct the data to the proper location. Files with different tags can be selected for transfer. Embodiments can be used with a defined backup system file replication process, such as present in the Data Domain File System. By using snapshots, incoming new data is continued to be ingested while the replication is in process and maintaining data consistency at the same time. This is achieved by performing operations on B+ Tree snapshots in conjunction with tag filtering on keys present in the leaf pages of these structures. This method efficiently makes a single pass walk of a B+ Tree in contrast with previous methods that look up files one-by-one via their pathname.

Granular data replication

A method of capturing a fractional snapshot of a distributed file system includes receiving a snapshot request to generate a factional snapshot of a file system. The request includes an indication of a subset of data of the file system. The method further includes identifying, by a processing device, the subset of data of the file system indicated by the snapshot request and generating the fractional snapshot comprising the subset of data of the file system indicated by the snapshot request.

Fractional consistent global snapshots of a distributed namespace

Embodiments for enabling snapshot backups in a global name space of a cluster network, by representing the name space of cluster network in an MTree, storing data files organized in a B+ Tree format on one or more data nodes, storing name specific information of the data files in a B+ Tree format in a meta node, wherein a B+ Tree of the meta node accesses each corresponding B+ Tree in each of the one or more data nodes. The process takes snapshot backups of individual MTree limbs, and links the limbs of each snapshot into groups based on a cluster identifier and snapshot identifier.

Global snapshot backups of a distributed name space

Disclosed herein are techniques for generating fractional checkpoints on a distributed file system by snapshotting subcomponents referred to as 'file sets.' The techniques include capturing a present time; calculating from configured parameters a future wakeup time for a subsequent set of checkpoints from the present time; walking a database of meta file set objects to locate at least one meta file set object; calculating a retention period for a snapshot that is being created; and generating a global data-less snapshot for the meta file set object and remote data file set objects associated with the meta file set object, and then repeating the process for subsequent file set objects in the database.

System and method of generating automatic checkpoints of a distributed file system

Systems and methods for aligning needs of virtual devices with hardware resources. The performance of virtual devices are tested using different groupings to determine mappings or relationships between the virtual devices and the physical devices from which they are drawn. Based on the results of the tests, spindle groups can be optimized.

Optimizing virtual storage devices by determining and optimizing associations between virtual storage devices and physical storage devices

Described is a system (and technique) to determine an appropriate path to a storage device and scheduling a data transfer by taking into consideration bandwidth, device performance, and the nature of the transfer. The system may use an efficient implementation to reduce overhead when making such a determination by utilizing a reservation table and selecting a dispatch that is efficient for a particular data transfer. For example, the system may determine a first-fit and a future-fit scheduling for an I/O request for each path to a storage device. The system may determine completion times for each of the types of scheduling for each of the paths, and accordingly, select the path and schedule with the earliest completion time.

Topology aware load optimized multipath I/O scheduler

Embodiments are directed to a requirements-based performance monitor (RBPM) that presents users, through a command line interface or graphical user interface, with a single number in the range of 1-100% and/or a color-coded indicator that allows users to readily tell if the I/O needs of their primary applications in a virtualized computing system are being sufficiently satisfied. The RBPM takes into account both device latency, throughput, IOPS, and slow I/O measurements and primary application requirements. The process uses detailed device latency tables for each device to allow a user to find the particular device and time that performance degradation occurred.

Growing community of inventors

Aptos, CA, United States of America

Charles J Hickey