Netapp, Inc.

Srinath Krishnamachari

Anshul Pundir

Ling Zheng

Dhaval Patel

Janice D'Sa

Sriranjani Babu

John A Muth

Chaitanya V Patel

Mandar Naik

A cluster-wide consistency checker ensures that two file systems of a storage input/output (I/O) stack executing on each node of a cluster are self-consistent as well as consistent with respect to each other. The file systems include a deduplication file system and a host-facing file system that cooperate to provide a layered file system of the storage I/O stack. The deduplication file system is a log-structured file system managed by an extent store layer of the storage I/O stack, whereas the host-facing file system is managed by a volume layer of the stack. Illustratively, each log-structured file system implements a key-value store and cooperates with other nodes of the cluster to provide a cluster-wide (global) key-value store. The consistency checker verifies and/or fixes on-disk structures of the layered file system to ensure its consistency. To that end, the consistency checker may determine whether there are inconsistencies in the key-value store and, if so, reconciles those inconsistencies from a client (volume layer) perspective.

Consistency checker for global de-duplication clustered file system

A layout of a transaction log enables efficient logging of metadata into entries of the log, as well as efficient reclamation and recovery of the log entries by a volume layer of a storage input/output (I/O) stack executing on one or more nodes of a cluster. The transaction log is illustratively a two stage, append-only logging structure, wherein the first level is non-volatile random access memory (NVRAM) embodied as a NVlog and the second stage is disk, e.g., solid state drive (SSD). During crash recovery, the log entries are examined for consistency and scanned to identify those entries that have completed and those that are active, which require replay. The log entries are walked from oldest to newest (using sequence numbers) searching for the highest sequence number. Partially complete log entries (e.g., log entries in-progress when a crash occurs) may be discarded for failing a checksum (e.g., a CRC error). Old value/new value logs may be used to implement roll-forward or roll-back semantics to replay the log entries and fix any on-disk data structures, first from NVRAM and then from on-disk logs.

Transaction log layout for efficient reclamation and recovery

A technique recovers from a low space condition associated with storage space reserved in an extent store to accommodate write requests received from a host and associated metadata managed by a layered file system of a storage input/output (I/O) stack executing on one or more nodes of a cluster. The write requests, including user data, are persistently recorded on non-volatile random access memory (NVRAM) prior to returning an acknowledgement to the host by a persistence layer of the storage I/O stack. Volume metadata managed by a volume layer of the layered file system is embodied as mappings from logical block addresses (LBAs) of a logical unit (LUN) accessible by the host to extent keys maintained by an extent store layer of the layered file system. Extent store metadata managed by the extent store layer is embodied as mappings from the extent keys to the storage locations of the extents on storage devices of storage arrays coupled to the nodes of the cluster. The space recovery technique accounts for storage space consumed in the extent store by user operations, i.e., write operations for the user data stored on the NVRAM at the persistence layer as well as the associated volume and extent store metadata, to ensure that the user data and associated metadata can be safely and reliably persisted in the extent store even during a low space condition.

Recovery from low space condition of an extent store

Embodiments herein are directed to efficient crash recovery of persistent metadata managed by a volume layer of a storage input/output (I/O) stack executing on one or more nodes of a cluster. Volume metadata managed by the volume layer is organized as a multi-level dense tree, wherein each level of the dense tree includes volume metadata entries for storing the volume metadata. When a level of the dense tree is full, the volume metadata entries of the level are merged with the next lower level of the dense tree. During a merge operation, two sets of generation IDs may be used in accordance with a double buffer arrangement: a first generation ID for the append buffer that is full (i.e., a merge staging buffer) and a second, incremented generation ID for the append buffer that accepts new volume metadata entries. Upon completion of the merge operation, the lower level (e.g., level 1) to which the merge is directed is assigned the generation ID of the merge staging buffer.

Growing community of inventors

Mountain View, CA, United States of America

Srinath Krishnamachari