International Business Machines Corporation

Saket Kumar

Venkateswara Rao Puvvada

Karrthik Kalaga Gopalakrishnan

Ashish Pandey

Karrthik K G

Ravi Kumar Komanduri

Saipriya M R

Ravindra Sure

Ashish Chaurasia

A method for minimizing data transfer and storage utilization on cloud object storage systems is disclosed. In one embodiment, such a method replicates a file from a production system to a cloud object storage system. The method determines whether a number of hard links associated with the file is greater than one. In the event the number is greater than one, the method creates, on the cloud object storage system, a special object for the file and associates the file with the special object. Upon creating a hard link on the production system in association with the file, the method replicates the hard link from the production system to the cloud object storage system without replicating data associated with the file. A metadata reference to the special object is added to the hard link on the cloud object storage system. A corresponding system and computer program product are also disclosed.

Minimizing data transfer and storage utilization on cloud object storage systems

Disclosed are techniques for multi-protocol multi-site file replication, including a method comprising capturing, at a gateway device, IO operations performed on a filesystem, the filesystem having inode bits. The method may further comprise enqueuing the captured IO operations onto a plurality of queues, wherein each of the plurality of queues is associated with one of a plurality of target sites; and independently managing replication of the IO operations to the target sites using the inode bits.

Multi-protocol multi-site replication

A computer-implemented method according to one embodiment includes receiving, on a first cluster site, a first I/O request to migrate a plurality of filesets from a second cluster site to the first cluster site. The first cluster site includes a plurality of gateway nodes. The method further includes identifying at least two of the gateway nodes having resources available to perform operations of the migration, and hashing information of a plurality of filesets against the identified gateway nodes. The information includes inode numbers of entities that are mounted during fulfillment of the first I/O request. Operations of the first I/O request are distributed to the identified gateway nodes based on the hashing, and the identified gateway nodes are instructed to fulfill the operations.

Hashing information of an input/output (I/O) request against a plurality of gateway nodes

A computer-implemented method includes migrating an item to a cache cluster from a home cluster in a storage system in a cutover process and initiating a data validation process with the home cluster. The method includes enabling at least one tunable parameter on the item on the cache cluster in the storage system. The tunable parameter is configured to adjust the data validation process with the home cluster. The tunable parameter includes at least one condition. The method includes enabling at least one flag on the item on the cache cluster in the storage system. The flag is configured to indicate that the at least one condition of the tunable parameter on the item in the storage system is met. In response to the at least one condition of the tunable parameter being met, the method includes ending the data validation process with the home cluster in the storage system.

Migration of uncached data after cutover

Provided are techniques for role reversal of primary and secondary sites with minimal replication delay. During a backup phase, an Input/Output (I/O) operation selected from a group comprising a create operation and a rename operation is received with one or more local attributes of a local file. The I/O operation is processed to generate one or more remote attributes. The one or more local attributes are stored with the one or more remote attributes. During a reversed phase, a write operation for the remote file is processed, and, the write operation is sent to the primary replication system with the one or more local attributes, where the primary replication system uses a local file handle in the one or more local attributes to access the local file for processing the write operation to sync the local file with the remote file.

Role reversal of primary and secondary sites with minimal replication delay

Provided are a computer program product, system, and method receiving at a cache node notification of changes to files in a source file system served from a cache file system at the cache node. A cache file system is established at the cache node as a local share of a source file system at the source node. The source node establishes a local share of the cache file system at the cache node. Notification is received, from the source node, that the source node modified a source control file for a source file at the source node. In response to receiving the notification, a cache control file, for a cached file in the cache file system, is updated to indicate the source file at the source node is modified. A request is sent to the source node to obtain data for the source file indicated as modified.

Receiving at a cache node notification of changes to files in a source file system served from a cache file system at the cache node

Provided a computer program product, system, and method for notifying a cache file system of changes to files in a source file system served from the cache file system. The source node establishes a local share of a cache file system at the cache node. The cache file system at the cache node comprises a local share of a source file system at the source node. In response to modifying a source file in the source file system, a source control file for the source file in the local share of the cache file system is updated to indicate the source file is modified. In response to updating the source control file, sending, to the cache node, indication of the updating of the source control file to cause the cache node to update a cache control file to indicate a change to the source file at the source node.

Notifying a cache file system of changes to files in a source file system served from the cache file system

A computer-implemented method according to one embodiment includes receiving, on a first cluster site, extended attributes associated with a first data operation where the first data operation was previously performed on data of a filesystem of second cluster site. In response to a second data operation being performed on data of a filesystem of the first cluster site, extended attributes associated with the second data operation are stored to a predetermined file of the filesystem of the first cluster site. The method further includes outputting, to the second cluster site, the extended attributes stored to the predetermined file of the filesystem of the first cluster site, and in response to a determination that a predefined event has occurred on the second cluster site, using the received extended attributes to fulfill a third data operation.

Exchanging extended attributes between different cluster sites in a clustered filesystem environment

A computer-implemented method according to one embodiment includes initializing a new gateway node at a first site in response to a failure of an old gateway node at the first site; creating, at the new gateway node, a list of dirty directories at the first site; synchronizing, with a second site by a background thread at the first site, all directories within the list of dirty directories; and synchronizing, with the second site by a foreground thread at the first site, new operations performed within the first site after the initialization of the new gateway node.

Recovering from a failure of an asynchronous replication node

A computer-implemented method according to one embodiment includes, in response to a determination that a predetermined operation has been performed on an object of a first file stored on a first cluster site, storing predetermined information about the object of the first file stored on the first cluster site. The predetermined information is stored on an extended attribute of the first file stored on the first cluster site. In response to a determination that the predetermined operation is performed on an object of a first file stored on a second cluster site, the predetermined information is removed from the extended attribute of the first file stored on the first cluster site. In response to a determination that a failure event has occurred on a queue of the first cluster site, a predetermined recovery process is performed, thereby enabling fulfillment of entries of the queue of the first cluster site.

Recovery from a clustered file system queue failure event using a modified extended attribute of a file

A computer-implemented method, according to one approach, includes: receiving a data operation request which includes an activated compound operation flag. The data operation request is added to a queue in a gateway node, and the data operation request is eventually transmitted to a disaster recovery site. An inode entry which corresponds to the portion of data is locked, and metadata associated with the inode entry is updated to indicate that the data operation request has been performed at the disaster recovery site. Supplemental data operation requests which correspond to the portion of data are also identified by evaluating the metadata associated with the inode entry. These supplemental data operation requests are transmitted to the disaster recovery site, and the metadata associated with the inode entry is updated to indicate that the supplemental data operation requests have been performed at the disaster recovery site. Furthermore, the inode entry is unlocked.

Using inode entries to mirror data operations across data storage sites

A system identifies (I) a first independent filesystem that (i) includes a file and (ii) uses file-handles to locate files and (II) a second independent filesystem that (i) operates independently of file-handles, associated with the first independent filesystem, when locating files and (ii) includes a copy of the file. The system generates a file-handle for the copy of the file on an inode of first independent filesystem based on a full-path for the copy of the file on the second independent filesystem. The system replicates one or more data blocks between the first independent filesystem and the second independent filesystem based, at least in part, on the full-path.

Continuous asynchronous replication from on-premises storage to cloud object stores

A computer-implemented method, according to one embodiment, includes: receiving data replication operations from one or more compute nodes at a primary location, storing the received data replication operations in a queue, and dividing the queued data replication operations into a number of independent sub-groups. The number of independent sub-groups is equal to or less than a number of worker gateway nodes. Each of the independent sub-groups are further added to a respective one of the worker gateway nodes. One or more instructions are also sent to each of the worker gateway nodes to send the data replication operations included in the respective independent sub-group to a remote storage location. Other systems, methods, and computer program products are described in additional embodiments.

Scalable and balanced distribution of asynchronous operations in data replication systems

A computer-implemented method according to one embodiment includes receiving, on a first gateway node of a first cluster site, an input/output (I/O) request for a first fileset, and determining a collection of mount operations in which to add the I/O request for the first fileset. The determining includes hashing information of the first fileset against more than one collection of mount operations. The method further includes adding the I/O request for the first fileset to a first determined collection of mount operations, and outputting the first determined collection of mount operations to a first network file system (NFS) server. The first NFS server is one of a plurality of NFS servers of a second cluster site. Moreover, the first NFS server and at least one other NFS server of the plurality of NFS servers are configured to receive a collection of mount operations from the first gateway node.

Growing community of inventors

Bettiah, India

Saket Kumar