Netapp, Inc.

Parag Sarfare

Ananthan Subramanian

Nikhil Mattankot

Asif Imtiyaz Pathan

Santhosh Selvaraj

Sandeep Yadav

Kevin Daniel Varghese

Szu-Wen Kuo

Rajesh Khandelwal

Suhas Urkude

Stephen Wu

Yong Cho

Amit Borase

Sudheer Kumar Vavilapalli

Harihara Kadayam

Hrishikesh Keremane

Kalaivani Arumugham

Vaiapuri Ramasubramaniam

Chaitanya V Patel

Prachi Deshmukh

Prachi Deskmukh

Manan Dahyabhai Patel

Joseph Brown

Tabriz Holtz

Anuja Jaiswal

Joanne Mikkelson

Travis Ryan Grusecki

Chaitanya Patel

Yong Eun Cho

Techniques are provided for implementing a persistent key-value store for caching client data, journaling, and/or crash recovery. The persistent key-value store may be hosted as a primary cache that provides read and write access to key-value record pairs stored within the persistent key-value store. The key-value record pairs are stored within multiple chains in the persistent key-value store. Journaling is provided for the persistent key-value store such that incoming key-value record pairs are stored within active chains, and data within frozen chains is written in a distributed manner across distributed storage of a distributed cluster of nodes. If there is a failure within the distributed cluster of nodes, then the persistent key-value store may be reconstructed and used for crash recovery.

Persistent key-value store and journaling system

A data management system can include a disk unit and a set of controllers. The disk unit can contain, at least in part, a set of storage media, a first persistent memory, and a second persistent memory. The set of storage media can be configured to implement a storage space. The set of controllers can be configured to write to the storage space and to implement a set of nodes including a first node and a second node. The first node can be configured to generate and write first node journal data to the first persistent memory. The second node can be configured to obtain a failure indication for the first node, obtain the first node journal data from the second persistent memory, and generate and provide a reply to a backend using the first node journal data.

Systems and methods for data management using a disk unit

Methods and systems for co-locating journaling and data storage are provided. Separate journal and volume partitions may be maintained within each logical storage unit (e.g., Logical Unit Number (LUN)) of a distributed storage system. Journaling of metadata associated with write requests received from one or more clients may be distributed by identifying a destination logical storage unit to which data associated with a given write request is to be stored and causing the data and metadata to be persisted to disk by journaling the metadata and the data to respective portions of an active log within the journal partition of the destination logical storage unit. By using the same logical storage unit for both journaling of write requests and writing the data associated with such write requests, the bottleneck due to there being only a single device or storage unit handling all metadata for all write requests can be avoided.

Co-located journaling and data storage for write requests

Failover methods and systems for a networked storage environment are provided. In one aspect, a read request associated with a first storage object is received, during a replay of entries of a log stored in a non-volatile memory of a second storage node for a failover operation initiated in response to a failure at a first storage node. The second storage node operates as a partner node of the first storage node. The read request is processed using a filtering data structure that is generated from the log prior to the replay and identifies each log entry. The read request is processed when the log does not have an entry associated with the read request, and when the filtering data structure includes an entry associated with the read request, the requested data is located at the non-volatile memory.

Network storage failover systems and associated methods

A data management system can include a set of storage media configured to implement a storage space and a set of controllers. The set of controllers can be configured to write to the storage space and to implement a set of nodes. The set of controllers can include a first controller that implements a first node and includes a first persistent memory, a second controller that implements a second node and includes a second persistent memory and a third controller that implements a third node and includes a third persistent memory. The third node can be configured to write third node journal data to the first persistent memory. The first node can be configured to generate first node journal data based on a first request received from a backend, write the first node journal data to the first persistent memory, and replicate the journal data to the second persistent memory.

Systems and methods for data management using controllers

Techniques are provided for a recovery process with selective ordering and concurrent operations in order to recover from a failure. Representations of active log structures are rebuilt within memory according to ordering values assigned to I/O operations logged within the active log structures. Representation of certain active log structures may be concurrently rebuilt based upon the active log structures comprising I/O operations that are non-overlapping within a distributed file system, have no dependencies, relate to different services, and/or target independent files. Representation of stale log structures are concurrently rebuilt within memory. While rebuilding the log structures and executing the I/O operations, a key value map is concurrently rebuilt within the memory for locating data of the I/O operations. Concurrent operations during the recovery process reduces the time to complete the recovery process, and thus reduces client downtime during the recovery process.

Recovery mechanism with selective ordering and concurrent operations

A method and system for co-locating journaling and data storage based on write requests. A write request that includes metadata and data is received from a client. A logical storage unit for storing the metadata and the data is identified. The logical storage unit is divided into a journal partition and a volume partition. The journal partition includes a first log and a second log. Which of the first log and the second log is an active log and which of the first log and the second log is an inactive log are identified. The metadata is recorded in a first location in the active log and the data is recorded in a second location in the active log during a single I/O operation. A reply is sent to the client after the metadata and the data are recorded in the journal partition.

Failover methods and systems for a networked storage environment are provided. A filtering data structure and a metadata data structure are generated before starting a replay of a log stored in a non-volatile memory of a second storage node, during a failover operation initiated in response to a failure at a first storage node. The second storage node operates as a partner node of the first storage node to mirror at the log one or more write requests received by the first storage node prior to the failure, and data associated with the one or more write requests. The filtering data structure identifies each log entry and the metadata structure stores a metadata attribute of each log entry. The filtering data structure and the metadata structure are used for providing access to a logical storage object during the log replay from the second storage node.

Failover methods and systems for a networked storage environment are provided. A metadata data structure is generated, before starting a replay of entries at a log stored in a non-volatile memory of a second storage node, during a failover operation initiated in response to a failure at a first storage node. The second storage node operates as a partner node of the first storage node, and the metadata structure stores a metadata attribute of each log entry. Furthermore, the metadata attribute of each log entry is persistently stored. The persistently stored metadata attribute is used to respond to a read request received during the replay by the second storage node, while a write request metadata attribute of a write request is used for executing the write request received by the second storage node during the replay.

A system and method for handling multi-node failures in a disaster recovery cluster is provided. In the event of an error condition, a switchover operation occurs from the failed nodes to one or more surviving nodes. Data stored in non-volatile random access memory is recovered by the surviving nodes to bring storage objects, e.g., disks, aggregates and/or volumes into a consistent state.

Growing community of inventors

San Jose, CA, United States of America

Parag Sarfare