Dell Products Inc.

Emc IP Holding Company LLC

Austin Voecks

Evgeny Popovich

Oliver Chung

Michael Frank

Smitha Jaya Shankar

Jonathan Mark Walton

Ben Ellerby

The described technology is generally directed towards replicating a filesystem namespace from a source system such as a source node cluster to a target system such as a target node cluster. Directory transfer tasks processed at the source are sent to the target build a replica of a hierarchical namespace on a target cluster, in which the namespace contains the directories and files, with the file content temporarily absent. Files transfer tasks processed at the source are sent to the target to write the content into the files, generally after namespace creation is complete. Directory transfer is recursive, as one directory transfer task creates another directory transfer task for each of its child directories, and can be concurrent and sent as bulk transfer requests for efficiency in reducing overall latency overhead. File content transfer can transfer the content of multiple small files in a single transfer task request.

Avoiding directory contention in distributed filesystem replication

The described technology is generally directed towards efficient detection of data changes between different data representations, to enable replication of the detected data changes. A process is presented that compares different 'snapshot' representations of data, wherein the data may have changed between the different snapshots. The process can detect changed files a well as namespace changes that occur when a file or directory is deleted, added, or moved to a different location.

Efficient detection of data changes between different data representations

The described technology is generally directed towards resolving namespace collisions on a target system when applying incremental changes (e.g., between snapshots) as part of an incremental replication from a source system. Detected changes to a filesystem namespace include file link, directory move and file and directory unlink operations. The changes are processed to recognize and overcome failures and other issues, such as to not process an unlink before a corresponding link, which can otherwise result in accidental deletion of a file. Deferred operations are maintained for later processing, including deferred directory moves for directory rename failures, deferred unlink operations and name collision fixes. After initial processing, the deferred move list is iteratively processed to overcome directory rename failures, the deferred unlink list is processed after performing corresponding link operations, and the name fix list is processed to rename previously colliding names.

Resolving namespace collisions during filesystem incremental replication

The described technology is generally directed towards data movement from data storage clusters. A data movement job can be split into data movement tasks, and the data movement tasks can be distributed among multiple nodes of a data storage cluster. In order to distribute the tasks, a data store that identifies the tasks can be split into multiple segments, and different groups of nodes can be assigned affinities to process tasks identified in the different segments. Furthermore, the nodes of each group of nodes can be assigned different start positions within their storage segment, to reduce the likelihood of contention between nodes in connection with claiming a same task. Some of the tasks can be generated in an already claimed state, which also reduces contention and allows for more efficient operation.

Data movement from data storage clusters

The described technology is generally directed towards fault tolerant cluster data handling techniques, as well as devices and computer readable media configured to perform the disclosed fault tolerant cluster data handling techniques. Nodes in a computing cluster can be configured to generate wire format resources corresponding to operating system resources. A wire format resource can comprise a cache key and a hint information to locate data, such as a file, corresponding to the operating system resource. The wire format resource can be stored in a resource cache along with a pointer that points to the operating system resource. The wire format resource can also be provided to client devices. Nodes in the computing cluster can be configured to receive and process client instructions that include wire format resources, as well as to use hint information to re-allocate data associated with a wire format resource.

Fault tolerant cluster data handling

Incremental replication between foreign system dataset stores is described herein. A method as described herein can include connecting to a target storage system to facilitate an initial transfer of a dataset from the system to the target storage system, wherein the system is distinct from the target storage system; in response to determining that a first version of the dataset is present on the target storage system, comparing the first version of the dataset to a second version of the dataset stored on the system; and, in response to determining that the second version of the dataset is more recent than the first version of the dataset, transferring a differential between the first version of the dataset and the second version of the dataset to the target storage system, the differential comprising less than all of the second version of the dataset.

Incremental replication between foreign system dataset stores

Systems and methods facilitating separation of dataset creation from movement in file replication are described herein. A method as described herein can include scheduling, by a first system operatively coupled to a processor, a first transfer of a dataset to a second system at a first time according to a first replication schedule; scheduling, by the first system, a second transfer of the dataset to a third system at the first time according to a second replication schedule; creating, by the first system, a snapshot of the dataset at a second time that is distinct from the first time according to a snapshot schedule that is distinct from the first replication schedule and the second replication schedule; and transferring, by the first system, at least a portion of the snapshot to the second system and the third system at the first time.

Growing community of inventors

Bainbridge, WA, United States of America

Austin Voecks