Netapp, Inc.

Christopher Cason

Daniel David McCarthy

Ananthan Subramanian

Arindam Banerjee

Ravikanth Dronamraju

Austino Nicholas Longo

Abdul Basit

Christopher Clark Corey

Bharadwaj Vellore Ramesh

Sneheet Kumar Mishra

Jian Hu

Mark David Olson

James Philip Wittig

Jared Cantwell

Christopher Alan Busick

Forrest Trimbell

Mohinish Vinnakota

Kyle Gene Knutson

Ankit Kumar

Avesh Kumar Agarwal

Brendan Kerr

Daniel Mccarthy

Techniques are provided for upgrading an external distributed storage layer that provides storage services to containerized applications hosted within a container hosting platform. An operator within the container hosting platform is custom configured to orchestrate, from within the container hosting platform, the upgrade for the external distributed storage layer. Because the external distributed storage layer and the container hosting platform are separate computing environment that utilize different namespaces, semantics, operating states, and/or application programming interfaces, a cluster controller within the container hosting platform is custom configured to reformat/translate commands between the external distributed storage layer and the container hosting platform for performing the upgrade. Because the external distributed storage layer upgrade may be part of an overall upgrade that upgrades the containerized applications hosted within the container hosting platform, the operator and cluster controller provide a single upgrade orchestration point for perform both upgrades in an orchestrated manner.

External distributed storage layer upgrade

Techniques are provided for repairing a primary slice file, affected by a storage device error, by using one or more dead replica slice files. The primary slice file is used by a node of a distributed storage architecture as an indirection layer between storage containers (e.g., a volume or LUN) and physical storage where data is physically stored. To improve resiliency of the distributed storage architecture, changes to the primary slice file are replicated to replica slice files hosted by other nodes. If a replica slice file falls out of sync with the primary slice file, then the replica slice file is considered dead (out of sync) and could potentially comprise stale data. If a storage device error affects blocks storing data of the primary slice file, then the techniques provided herein can repair the primary slice file using non-stale data from one or more dead replica slice files.

Slice file recovery using dead replica slice files

In various examples, data storage is managed using a distributed storage management system that is resilient. Data blocks of a logical block device may be distributed across multiple nodes in a cluster. The logical block device may correspond to a file system volume associated with a file system instance deployed on a selected node within a distributed block layer of a distributed file system. Each data block may have a location in the cluster identified by a block identifier associated with each data block. Each data block may be replicated on at least one other node in the cluster. A metadata object corresponding to a logical block device that maps to the file system volume may be replicated on at least another node in the cluster. Each data block and the metadata object may be hosted on virtualized storage that is protected using redundant array independent disks (RAID).

Distributed file system that provides scalability and resiliency

Techniques are provided for dynamically implementing quality of service policies for a distributed storage system based upon resources saturation. A quality of service policy is defined for throttling I/O operations received by a node of the distributed storage system based upon whether resources of the node have become saturated. The quality of service policy is dynamically implemented based upon ever changing resource utilization and saturation. Dynamically implementing the quality of service policy improves the ability to efficiently utilize resources of the node compared to conventional static polices that cannot adequately react to such changing considerations and resource utilization/saturation. With conventional static policies, an administrator manually defines a minimum amount of guaranteed resources and/or a maximum resource usage cap that could be set to values that result in inefficient operation and resource starvation. Dynamically implementing the quality of service policy results in more efficient operation and mitigates resource starvation.

Dynamic quality of service implementation based upon resource saturation

Techniques are provided for dynamically implementing quality of service policies using a configurable quality of service provider pipeline. A quality of service policy is defined for throttling I/O operations received by a node based upon whether resources of the node have become over utilized. The quality of service policy is used to dynamically construct a quality of service provider pipeline with select quality of service providers that improve the ability to efficiently utilize resources compared to conventional static polices that cannot adequately react to changing considerations and resource utilization/saturation. With conventional static policies, an administrator manually defines a minimum amount of guaranteed resources and/or a maximum resource usage cap that could be set to values that result in inefficient operation and resource starvation. Dynamically constructing and utilizing the quality of service provider pipeline results in more efficient operation and mitigates resource starvation.

Configurable quality of service provider pipeline

Systems and methods for scaling application and/or storage system functions of a distributed storage system based on a heterogeneous resource pool are provided. According to one embodiment, the distributed storage system has a composable, service-based architecture that provides scalability, resiliency, and load balancing. The distributed storage system includes a cluster of nodes each potentially having differing capabilities in terms of processing, memory, and/or storage. The distributed storage system takes advantage of different types of nodes by selectively instating appropriate services (e.g., file and volume services and/or block and storage management services) on the nodes based on their respective capabilities. Furthermore, disaggregation of these services, facilitated by interposing a frictionless layer (e.g., in the form of one or more globally accessible logical disks) therebetween, enables independent and on-demand scaling of either or both of application and storage system functions within the cluster while making use of the heterogeneous resource pool.

Dynamically scaling application and storage system functions based on a heterogeneous resource pool available for use by a distributed storage management system

A distributed storage management system comprising nodes that form a cluster, a distributed block layer that spans the nodes in the cluster, and file system instances deployed on the nodes. Each file system instance comprises a data management subsystem and a storage management subsystem disaggregated from the data management subsystem. The storage management subsystem comprises a node block store that forms a portion of the distributed block layer and a storage manager that manages a key-value store and virtualized storage supporting the node block store. A file system volume hosted by the data management subsystem maps to a logical block device hosted by the virtualized storage in the storage management subsystem. The key-value store includes, for a data block of the logical block device, a key that comprises a block identifier for the logical block device and a value that comprises the data block.

A technique provides efficient data protection, such as erasure coding, for data blocks of volumes served by storage nodes of a cluster. Data blocks associated with write requests of unpredictable client workload patterns may be compressed. A set of the compressed data blocks may be selected to form a write group and an erasure code may be applied to the group to algorithmically generate one or more encoded blocks in addition to the data blocks. Due to the unpredictability of the data workload patterns, the compressed data blocks may have varying sizes. A pool of the various-sized compressed data blocks may be established and maintained from which the data blocks of the write group are selected. Establishment and maintenance of the pool enables selection of compressed data blocks that are substantially close to the same size and, thus, that require minimal padding.

Pooling blocks for erasure coding write groups

Systems and methods for quality of service management are provided. According to one embodiment, a non-transitory computer-readable medium comprises instructions that when executed by the processing resource cause the processing resource to receive, in a normalizing agent, one or more compute load parameters from one or more background compute processes executing on the one or more computer systems and one or more Quality of Service (QoS) parameters for one or more client compute processes executing on the one or more computer systems, convert the one or more compute load parameters to one or more normalized utilization metrics, and execute instructions that cause a processor to adjust a compute resource allocation dedicated to the one or more background compute processes based at least in part on the one or more normalized utilization metrics and the one or more QoS parameters.

Centralized quality of service management

A technique manages bandwidth allocated among input/output operations, such as reads and writes, to storage devices coupled to storage nodes of a cluster. The technique balances the writes in a manner that reduces latency of reads, while allowing the writes to complete in a desired amount of time. The writes include write types, such as client writes, data migration writes, block transfer writes, and recycling writes, which are defined by differing characteristics and relative priorities. To ensure timely completion of the write types, the technique provides periodic time intervals over which the writes may be balanced and allocated sufficient bandwidth to access the storage devices. The time intervals may include shuffle intervals within a larger distribution interval. In addition, the technique throttles certain write types at the storage device level to maintain consistent read performance. Throttling is based on a credit system that allocates bandwidth as 'credits' based on write type.

Write type based crediting for block level write throttling to control impact to read input/output operations

A technique is configured to maintain multiple copies of data served by storage nodes of a cluster during upgrade of a storage node to ensure continuous protection of the data served by the nodes. The data is logically organized as one or more volumes on storage devices of the cluster and includes metadata that describe the data of each volume. A data protection system may be configured to maintain at least two copies of the data in the cluster during upgrade to a storage node that is assigned to host one of the copies of the data but that is taken offline during the upgrade. As a result, an original slice service of the node may be rendered unavailable during the upgrade. In response, the technique redirects replicated data targeted to the original slice service to a standby pool of slice services in accordance with a degraded redundant metadata service of the cluster. In the event the standby slice service itself subsequently becomes unavailable, another standby slice service from the standby pool is activated to receive the subsequent data. In this manner, cascading failure of secondary slice slices is handled.

Standby copies withstand cascading fails

A technique is configured to reduce an amount of memory (i.e., memory footprint) usage by each storage node of a cluster needed to store metadata while providing fast and efficient servicing of data in accordance with storage requests issued by a client of the cluster. Illustratively, a block identifier (ID) is used to identify a block of data serviced by the storage node. Metadata embodied as mappings between block IDs and locations of data blocks in the cluster are illustratively maintained in map fragments. A map fragment may be embodied as 'active' map fragment or a 'frozen' map fragment. An active map fragment refers to a map fragment that has space available to store a mapping, whereas a frozen map fragment refers to a map fragment that is full, i.e., has no available space for storing a mapping. In order to reduce the memory footprint of each storage node, yet still provide fast and efficient servicing of data by the node, the active map fragments are preferably maintained in memory as &#x201c;in-core&#x201d; data structures, whereas the frozen map fragments are paged-out and stored on storage devices of the cluster as &#x201c;on-disk&#x201d; map fragment structures.

Growing community of inventors

Boulder, CO, United States of America

Christopher Cason