Netapp, Inc.

Oath Inc.

Anil Paul Thoppil

Cheryl Marie Thompson

Qinghua Zheng

Ananthan Subramanian

Kevin Daniel Varghese

Wei Sun

Ganga Bhavani Kondapalli

Sridevi Jantli

Ronak Girishbhai Ghadiya

Nicholas Gerald Zehender

Jeevan Hunsur Eswara

Santhosh Selvaraj

Meera Odugoudar

Anne Maria Vasu

Szu-Wen Kuo

Arindam Banerjee

Srishylam Simharajan

Manan Dahyabhai Patel

James Leong

Vitaly Revsin

Vani Vully

Nikul Y Patel

Vikhyath Rao

Raymond Jordan Go

Sanjay Subramanian Seshadri

Mark David Olson

Sunitha Sunil Sankar

Garima Choudhary

Anne Marie Vasu

Systems and methods are described for a streamlined garbage collection process during which data integrity checking is also performed for a distributed key-value (KV) store utilized by a distributed storage system. According to one embodiment, by making use of full or truncated block IDs (rather than an intermediate probabilistic data structure, such as a Bloom filter) for garbage collection, data integrity checking can be performed concurrently almost for free. During garbage collection, a block ID compare list may be compared to block IDs within the distributed KV store. If a particular block ID is present in the distributed KV store but is missing from the block ID compare list, the corresponding data block represents garbage to be collected. If the particular block ID is present in the block ID compare list but missing from the distributed KV store, a data integrity error has been identified.

Combined garbage collection and data integrity checking for a distributed key-value store

Systems and methods that make use of cluster-level redundancy within a distributed storage system to address various node-level error scenarios are provided. Rather than making use of a generalized one-size-fits-all approach in an effort to reduce complexity, an approach tailored to the node-level error scenario at issue may be performed to avoid doing more than necessary. According to one embodiment, responsive to identification of a failed RAID stripe by a node of a cluster of a distributed storage management system, for each block ID of multiple block IDs associated with the failed RAID stripe, a data block is restored corresponding to the block ID by reading the data block from another node of the cluster having a redundant copy of the data block; and writing the redundant copy of the data block to a storage area of the node that is unaffected by the failed RAID stripe.

Use of cluster-level redundancy within a cluster of a distributed storage management system to address node-level errors

Techniques are provided for key-value store and file system integration to optimize key value store operations. A key-value store is integrated within a file system of a node. A log structured merge tree of the key-value store may be populated with a key corresponding to a content hash of a value data item stored separate from the key. A random distribution search may be performed upon a sorted log of the log structured merge tree to identify the key for accessing the value data item. A starting location for the random distribution search is derived from key information, a log size of the sorted log, and/or a keyspace size of a keyspace associated with the key.

Key-value store and file system integration

Techniques are provided for object store mirroring. Data within a storage tier of a node may be determined as being data to tier out to a primary object store based upon a property of the data. A first object is generated to comprise the data. A second object is generated to comprise the data. The first object is transmitted to the primary data store for storage in parallel with the second object being transmitted to a mirror object store for storage. Tiering of the data is designated as successful once acknowledgements are received from both the primary object that the first object was stored and the mirror object store that the second object was stored.

Object store mirroring based on checkpoint

Techniques are provided for implementing a hash building process and an append hash building process. The hash building process builds in-memory hash entries for bins of keys stored within sorted logs of a log structured merge tree used to store keys of a key-value store. The in-memory hash entries can be used to identify the starting locations of bins of keys within the log structured merge tree so that a key within a bin can be searched for from the starting location of the bin as opposed to having to search the entire log structured merge tree. The append hash building process builds two hashes that can be used to more efficiently locate keys and/or ranges of keys within an unsorted append log that would otherwise require a time consuming binary search of the entire append log.

In-memory hash entries and hashes used to improve key search operations for keys of a key value store

Systems and methods that make use of cluster-level redundancy within a distributed storage management system to address various node-level error scenarios are provided. Rather than using a generalized one-size-fits-all approach to reduce complexity, an approach tailored to the node-level error scenario at issue may be performed to avoid doing more than necessary. According to one embodiment, after identifying a missing branch of a tree implemented by a KV store of a first node of a cluster of a distributed storage management system, a branch resynchronization process may be performed, including, for each block ID in the range of block IDs of the missing branch (i) reading a data block corresponding to the block ID from a second node of the cluster that maintains redundant information relating to the block ID; and (ii) restoring the block ID within the KV store by writing the data block to the first node.

Techniques are provided for implementing a garbage collection process and a prediction read ahead mechanism to prefetch keys into memory to improve the efficiency and speed of the garbage collection process. A log structured merge tree is used to store keys of key-value pairs within a key-value store. If a key is no longer referenced by any worker nodes of a distributed storage architecture, then the key can be freed to store other data. Accordingly, garbage collection is performed to identify and free unused keys. The speed and efficiency of garbage collection is improved by dynamically adjusting the amount and rate at which keys are prefetched from disk and cached into faster memory for processing by the garbage collection process.

Prefetching keys for garbage collection

Systems and methods for performing a zero-copy volume move between nodes of a distributed storage system are provided. In one example, an approach for performing a zero-copy volume move is proposed in which volume data may be maintained in place within a storage pod and need not be copied to move a given volume between the source node and the destination node. In one embodiment, metadata (e.g., a top-most physical volume block number (PVBN) of a node tree representing the volume at issue) of a write-anywhere file system is copied from the source node to the destination node. Since the storage pod is associated with a global PVBN space that is visible and accessible to all nodes of the distributed storage system, as a result of copying the top-mode PVBN of the volume to the destination node, anything below the top-most PVBN will automatically be visible to the destination node.

Zero-copy volume move within a distributed storage system

Systems and methods that make use of cluster-level redundancy within a distributed storage management system to address various node-level error scenarios are provided. According to one embodiment, a first node of multiple nodes of distributed storage system represented in a form of a cluster of the multiple of nodes, identifies the potential existence of an error associated with a Redundant Array of Independent Disks (RAID) stripe. A list of block identifiers (IDs) associated with the RAID stripe may then be identified. Rather than performing a traditional RAID recovery/reconstruction approach that is resource intensive in nature and that requires an excessive amount of rebuild time, a more efficient RAID stripe resynchronization process may be performed to restore data associated with the RAID stripe.

Techniques are provided for multi-tier write allocation. A storage system may store data within a multi-tier storage environment comprising a first storage tier (e.g., storage devices maintained by the storage system), a second storage tier (e.g., a remote object store provided by a third party storage provider), and/or other storage tiers. A determination is made that data (e.g., data of a write request received by the storage system) is to be stored within the second storage tier. The data is stored into a staging area of the first storage tier. A second storage tier location identifier, for referencing the data according to a format utilized by the second storage tier, is assigned to the data and provided to a file system hosting the data. The data is then destaged from the staging area into the second storage tier, such as within an object stored within the remote object store.

Multi-tier write allocation

Techniques are provided for orphan object detection, invalid sequence number detection, and asynchronous object cleanup. A storage system may store data within one or more tiers of storage, such as a storage tier (e.g., solid state storage and disks maintained by the storage system), a remote object store (e.g., storage provided by a third party storage provider), and/or other storage tiers. Orphan objects, within the remote object store, that are no longer used by the storage system may be detected and/or deleted. When an aggregate of volumes is deleted, corresponding objects, within the remote object store, may be identified and/or deleted. Invalid sequence numbers (e.g., lost or corrupt sequence numbers locally maintained in a metafile) assigned to objects within the remote object store may be identified, deleted, and/or fixed.

Object and sequence number management

Systems and methods are described for a streamlined garbage collection process during which data integrity checking is also performed for a distributed key-value (KV) store utilized by a distributed storage management system. According to one embodiment, by making use of full or truncated block IDs (rather than an intermediate probabilistic data structure, such as a Bloom filter) for garbage collection, data integrity checking can be performed concurrently almost for free. During garbage collection, a block ID compare list is compared to block IDs within the distributed KV store. If a particular block ID is present in the distributed KV store but is missing from the block ID compare list, the corresponding data block represents garbage to be collected. If the particular block ID is present in the block ID compare list but missing from the distributed KV store, a data integrity error has been identified.

Systems and methods that make use of cluster-level redundancy within a distributed storage management system to address various node-level error scenarios are provided. Rather than making use of a generalized one-size-fits-all approach in an effort to reduce complexity, an approach tailored to the node-level error scenario at issue may be performed to avoid doing more than necessary. According to one embodiment, responsive to identification of a failed RAID stripe by a node of a cluster of a distributed storage management system, for each block ID of multiple block IDs associated with the failed RAID stripe, a data block is restored corresponding to the block ID by reading the data block from another node of the cluster having a redundant copy of the data block; and writing the redundant copy of the data block to a storage area of the node that is unaffected by the failed RAID stripe.

Techniques are provided for implementing a defragmentation process during a merge operation performed by a re-compaction process upon a log structured merge tree. The log structured merge tree is used to store keys of key-value pairs within a key-value store. As the log structured merge tree fills with keys over time, the re-compaction process is performed to merge keys down to lower levels of the log structured merge tree to re-compact the keys. Re-compaction can result in fragmentation because there is a lack of spatial locality of where the re-compaction operations re-writes the keys within storage. Fragmentation increases read and write amplification when accessing the keys stored in different locations within the storage. Accordingly, the defragmentation process is performed during a last merge operation of the re-compaction process in order to store keys together within the storage, thus reducing read and write amplification when accessing the keys.

Defragmentation for log structured merge tree to improve read and write amplification

Techniques are provided for remote object store error handling. A storage system may store data within one or more tiers of storage, such as a local storage tier (e.g., solid state storage and disks maintained by the storage system), a remote object store (e.g., storage provided by a third party storage provider), and/or other storage tiers. Because the remote object store may not provide the same data consistency and guarantees that the storage system provides for clients such as through the local storage tier, additional validation is provided by the storage system for the remote object store. For example, when data is put into an object of the remote object store, a verification get operation is performed to read and validate information within a header of the object. Other verifications and checks are performed such as using a locally stored metafile to detect corrupt or lost metadata and/or objects.

Growing community of inventors

Pleasanton, CA, United States of America

Anil Paul Thoppil