Netapp, Inc.

Anoop Chakkalakkal Vijayan

Akhil Kaushik

Sohan Shetty

Dhruvil Shah

Murali Subramanian

Omprakash Khandelwal

Arun Kumar Selvam

Krishna Murthy Chandraiah Setty Narasingarayanapeta

Shrey Sengar

Prasad Srinivas

Arul Valan

Balaji Ramani

Harsha Handral Sridhara

Rakshit Karnawat

Ganesh Marappa

Preetham Kudgi Shenoy

Veena T S

Chinmoy Dey

Rohit Chaudhary

Sudhakar Jha

Reshmi G

Vineeth Kumar T

Systems and methods include negotiating a primary bias state for primary and secondary storage sites when a mediator is temporarily unavailable for a multi-site distributed storage system. In one example, a computer-implemented method comprises detecting, with the primary storage site having a primary storage cluster, a temporary loss of connectivity to a mediator or a failure of the mediator. The computer-implemented method includes negotiating the primary bias state and setting the primary bias state on a secondary storage cluster of the secondary storage site when the secondary storage cluster detects a temporary loss of connectivity to the mediator, determining whether the primary storage cluster receives a confirmation of the secondary storage cluster setting the primary bias state, and setting the primary bias state on the primary storage cluster when the primary storage cluster receives the confirmation.

Methods and systems for negotiating a primary bias state in a distributed storage system

Systems and methods are described for efficiently performing various operations at the granularity of a consistency group (CG) within a cross-site storage solution. An example of one of the various operations includes an independent and parallel resynchronization approach that independently brings individual volumes of a CG to a steady state of in-synchronization (InSync), thereby contributing to scalability of CGs by supporting CGs having a large number of member volumes without requiring a change to the resynchronization process. Another example includes preserving dependent write-order consistency when a remote mirror copy goes out-of-synchronization (OOS) for any reason by driving all member volumes OOS responsive to any member volume becoming OOS. Yet another example includes independent creation of snapshots by member volumes to support efficient and on-demand creation by an application of a common snapshots of all or a subset of peered member volumes of a CG with which the application is associated.

Performing various operations at the granularity of a consistency group within a cross-site storage solution

Systems and methods are described for efficiently processing events related to a relationship between a primary copy of data at a primary storage system and a mirror copy of the data at a cross-site secondary storage system of a multi-site distributed storage system. According to an example, a mediator agent that is configured on both primary and secondary storage systems provides coordination and serialization for various events generated in the relationship. The multi-site distributed storage system performs actions based on the event processing such as performing a failover operation from the primary storage system to the secondary storage system or resynchronizing the relationship to ensure application protection and availability.

Methods and systems to interface between a multi-site distributed storage system and an external mediator to efficiently process events related to continuity

Multi-site distributed storage systems and computer-implemented methods are described for improving a resumption time of input/output (I/O) operations during an automatic unplanned failover (AUFO). A computer-implemented method includes monitoring, with a second cluster, heartbeat information received at ultra-short time intervals from a first connection of one or more storage objects of the first cluster, determining, with the second cluster, whether the heartbeat information from the first connection is received during an ultra-short time interval, and intelligently routing heartbeat information from the one or more storage objects of the first cluster from the first connection to a second connection when the heartbeat information from the first connection is not received during the ultra-short time interval.

Methods and systems to improve input/output (I/O) resumption time during a non-disruptive automatic unplanned failover from a primary copy of data at a primary storage system to a mirror copy of the data at a cross-site secondary storage system

Multi-site distributed storage systems and computer-implemented methods are described for improving a resumption time of input/output (I/O) operations during an automatic unplanned failover (AUFO). A computer-implemented method includes determining, with a second storage cluster, whether heartbeat information from one or more storage objects of a CG of a first set of CGs is received during a time period, determining an out of sync state for a data replication relationship between the CG of the first set of CGs and a mirrored CG of a second set of CGs when the heartbeat information is not received during the time period and sending a single bulk role change call with a cluster identifier from the second cluster to an external mediator to provide a role change from follower to leader in the second set of CGs.

Methods and systems to improve input/output (I/O) resumption time by batching multiple non-conflicting operations during a non-disruptive automatic unplanned failover from a primary copy of data at a primary storage system to a mirror copy of the data at a cross-site secondary storage system

Systems and methods are provided for bringing a volume of a consistency group (CG) into an in-synchronization (InSync) state while other volumes of the CG remain in the InSync state. According to an example, in order to support recovery from disruptive events in a manner that ensures a zero recovery point objective (RPO) guarantee and insulates an application making use of the CG from adverse impacts, responsive to a triggering event, a Fast Resync process may first be attempted to promptly bring an affected volume back into an in-synchronization (InSync) state from an out of synchronization (OOS) state while allowing other members of the CG to remain in the InSync state. Should the Fast resync process be unsuccessful in bringing the volume back into the InSync state within a predetermined or configurable time threshold, then a second type of resynchronization process may be employed at the CG level.

Resynchronization of individual volumes of a consistency group (CG) within a cross-site storage solution while maintaining synchronization of other volumes of the CG

Multi-site distributed storage systems and computer-implemented methods are described for improving a resumption time for processing of input/output (I/O) operations during an automatic unplanned failover (AUFO). A first storage cluster includes a first set of consistency groups (CGs) and a second storage cluster includes a second mirrored set of CGs. A computer-implemented method includes prefetching, with a user space of the second storage cluster, configuration information from a replicated database prior to starting the AUFO workflow, sending the configuration information to a kernel space of the second storage cluster on a per CG level while queuing the AUFO workflow, and determining if any in progress workflows conflict with the AUFO workflow.

Methods and systems to improve resumption time of input/output (I/O) operations based on prefetching of configuration data and early abort of conflicting workflows during a non-disruptive automatic unplanned failover from a primary copy of data at a primary storage system to a mirror copy of the data at a cross-site secondary storage system

Systems and methods for making a cross-site storage solution resilient towards mediator unavailability are provided. According to one embodiment, a stretched storage system is operable to bring a mediator associated with a primary and secondary distributed storage system back into the role of an arbitrator for peered consistency groups (CGs). A mediator reseed status indicator is maintained for multiple CGs to identify when the mediator's status information for a CG is stale. When the mediator becomes available and a local CG is identified as the subject of a mediator reseed process, the master node of the primary that hosts a master copy of a dataset for the local CG performs the reseed process, including: (i) causing relationship status information for the local CG to be updated on the mediator to the current state maintained by the primary; and (ii) resetting the mediator reseed status indicator.

Reseeding a mediator of a cross-site storage solution

A configuration for a component of a primary node is synchronized with a configuration for a component of a partner node in a different cluster by replicating the primary node configuration with the partner node. A baseline configuration replication comprises a snapshot of a component configuration on the primary. The baseline configuration can be generated by traversing through the configuration objects, capturing their attributes and encapsulating them in a package. The baseline package can then be transferred to the partner node. The configuration objects can be applied on the partner node in the order in which they were captured on the primary node. Attributes of the configuration objects are identified that are to be transformed. Values for the identified attributes are transformed from a name space in the primary node to a name space in the partner node.

Synchronizing configuration of partner objects across distributed storage systems using transformations

Systems and methods are described for a non-disruptive planned failover from a primary copy of data at a primary storage system to a mirror copy of the data at a cross-site secondary storage system. According to an example, a planned failover feature of a multi-site distributed storage system provides an order of operations such that a primary copy of a first data center continues to serve I/O operations until a mirror copy of a second data center is ready. This planned failover feature improves functionality and efficiency of the distributed storage system by providing non-disruptiveness during planned failover&#x2014;even if various failures occur. The planned failover feature also includes a persistent fence to avoid serving I/O operations during a timing window when both primary data storage and secondary data storage are attempting to have a master role to serve I/O operations and this avoids a split-brain situation.

Methods and systems for a non-disruptive planned failover from a primary copy of data at a primary storage system to a mirror copy of the data at a cross-site secondary storage system

Techniques are provided for neutralizing replication errors. An operation is executed upon a first storage object and is replicated as a replicated operation for execution upon a second storage object. A first error may be received for the replicated operation. Instead of transitioning to an out of sync state and aborting the operation, a wait is performed until a result of the attempted execution of the operation is received. If the first error is the same as a second error returned for the operation, then the operation and replicated operation are considered successful and a synchronous replication relationship is kept in sync. If the first error and the second error are different errors, then an error response is returned for the operation and the synchronous replication relationship is transitioned to out of sync.

Method and apparatus to neutralize replication error and retain primary and secondary synchronization during synchronous replication

Systems and methods are described for a non-disruptive planned failover from a primary copy of data at a primary storage cluster to a mirror copy of the data at a cross-site secondary storage cluster without using an external mediator. According to an example, a planned failover feature of a multi-site distributed storage system provides an order of operations such that a primary copy of a first data center continues to serve I/O operations until a mirror copy of a second data center is ready. This planned failover feature improves functionality and efficiency of the distributed storage system by providing non-disruptiveness during planned failover without using an external mediator based on a primary storage cluster being selected as an authority to implement a state machine with a persistent configuration database to track a planned failover state for the planned failover.

Methods and systems for a non-disruptive planned failover from a primary copy of data at a primary storage system to a mirror copy of the data at a cross-site secondary storage system without using an external mediator

Growing community of inventors

Bengaluru, India

Anoop Chakkalakkal Vijayan