Netapp, Inc.

Tyler W Cady

Austino Nicholas Longo

Joseph Ray Thomas, Iii

Systems and methods are provided for learning normal behavior for user roles of an application running within a cluster of container orchestration platform and based thereon proactively taking action responsive to suspicious events. According to one embodiment, an event data stream is created by an API server of the cluster. The data for each event includes information regarding a request made to an API exposed by the API server with which the event is associated and a user of the application by which the event was initiated. The data is augmented with a role associated with the user and an anomaly threshold for the role. Normal behavior is learned by an ML algorithm of respective user roles by processing the augmented data. When an anomaly score associated with a particular event is output by the ML algorithm that exceeds the anomaly threshold, a predefined or configurable action is triggered.

Proactively taking action responsive to events within a cluster based on a range of normal behavior learned for various user roles

A system is described. The system includes a processing resource and a non-transitory computer-readable medium, coupled to the processing resource, having stored therein instructions that when executed by the processing resource cause the processing resource to receive a plurality of quality of service (QoS) parameters and client preferences from a client device and manage a QoS policy based on a plurality of QoS objectives included in the received QoS parameters, wherein the plurality of QoS objectives comprise input output operations per second (IOPS), throughput and latency.

Quality of service management mechanism

Systems and methods are described for using a Deep Reinforcement Learning (DRL) agent to automatically tune Quality of Service (QoS) settings of a distributed storage system (DSS). According to one embodiment, a DRL agent is trained in a simulated environment to select QoS settings (e.g., a value of one or more of a minimum IOPS parameter, a maximum IOPS parameter, and a burst IOPS parameter). The training may involve placing the DRL agent into every feasible state representing combinations of QoS settings, workload conditions, and system metrics for a period of time for multiple iterations, and rewarding the DRL agent for selecting QoS settings that minimize an objective function based on a selected measure of system load. The trained DRL agent may then be deployed to one or more DSSs to constantly update QoS settings so as to minimize the selected measure of system load.

Automatically tuning a quality of service setting for a distributed storage system with a deep reinforcement learning agent

Systems and methods for automated tuning of Quality of Service (QoS) settings of volumes in a distributed storage system are provided. According to one embodiment, one or more characteristics of a workload of a client to which a storage node of multiple storage nodes of the distributed storage system is exposed are monitored. After a determination has been made that a characteristic meets or exceeds a threshold, (i) information regarding multiple QoS settings assigned to a volume of the storage node utilized by the client is obtained, (ii) a new value of a burst IOPS setting of the multiple QoS settings is calculated by increasing a current value of the burst IOPS setting by a factor dependent upon a first and a second QoS setting of the multiple QoS settings, and (iii) the new value of the burst IOPS setting is assigned to the volume for the client.

Automated tuning of a quality of service setting for a distributed storage system based on internal monitoring

A system, method, and machine-readable storage medium for providing a quality of service (QoS) recommendation to a client to modify a QoS setting are provided. In some embodiments, a set of volumes of a plurality of volumes may be determined. Each volume of the set of volumes may satisfy a first QoS setting assigned to the volume and a second QoS setting assigned to the volume. The plurality of volumes may reside in a common cluster and may be accessed by the client. Additionally, a subset of the set of volumes may be determined. Each volume of the subset may satisfy an upper bound of a range based on a minimum IOPS setting of the volume. A QoS recommendation to the client to modify the first QoS setting may be transmitted for one or more volumes of the subset.

Quality of service (QOS) setting recommendations for volumes across a cluster

A system, method, and machine-readable storage medium for providing a recommendation to a client to modify minimum IOPS settings are provided. In some embodiments, a client may assign a minimum input/output operations per second (IOPS) setting to each volume of a plurality of volumes. The plurality of volumes may reside in a common cluster and include a first number of volumes. A set of volumes of the plurality of volumes may be determined, where each volume of the set of volumes has a relative disparity that satisfies a relative threshold. The set of volumes includes a second number of volumes. A minimum IOPS recommendation may be transmitted to the client to modify (e.g., increase or decrease) the minimum IOPS settings of the set of volumes in response to determining that a ratio of the second number of volumes to the first number of volumes is greater than a performance threshold.

Disparity of quality of service (QoS) settings of volumes across a cluster

Systems and methods for automated tuning of Quality of Service (QoS) settings of volumes in a distributed storage system are provided. According to one embodiment, responsive to a predetermined event, information regarding a multiple QoS settings assigned to a volume of a distributed storage system that is being utilized by a client are obtained. A difference between a first QoS setting of the multiple QoS settings and a second QoS setting of the multiple QoS settings is determined. Responsive to determining the difference is less than a threshold a new value of the first QoS setting or a third QoS setting of the multiple QoS settings that is greater than a respective current value of the first QoS setting or the third QoS setting is determined and assigned to the volume for the client.

Systems and methods for enhancing the representation of outliers in a distribution of telemetry data of a monitored system are provided. According to one embodiment, telemetry data of the monitored system may be continuously collected. Frequency values representing a frequency of occurrence of corresponding telemetry data of the collected telemetry data may be generated by aggregating the collected telemetry data. As the vast majority of telemetry data is expected to represent a normal operating state of the system and relatively few, if any, of the telemetry data (e.g., outliers) will be indicative of one or more events of significance, the resulting distribution of the frequency values is highly skewed. In order to facilitate visualization of the distribution that accentuates the outliers, display characteristics may be calculated for the frequency values by applying a visualization model based on a weighted combination of multiple data transformations to each of the frequency values.

Visualization of outliers in a highly-skewed distribution of telemetry data

A system, method, and machine-readable storage medium for providing a set of recommended quality of service (QoS) settings are provided. In some embodiments, providing the recommendation includes receiving a set of QoS settings of a volume for a client, a set of measured QoS metrics of the volume for the client, and a measure of load for a slice service corresponding to the volume. Providing the recommendation further includes determining a predicted QoS metric of the volume and a predicted load of the slice service. Providing the recommendation also includes determining, based on the predicted QoS metric, the predicted load, and the set of QoS settings, a set of recommended QoS settings to the client. The set of QoS settings of the volume for the client is then updated with the set of recommended QoS settings.

Quality of service (QoS) settings of volumes in a distributed storage system

Systems and methods are described for using a Deep Reinforcement Learning (DRL) agent to automatically tune Quality of Service (QoS) settings of volumes in a distributed storage system (DSS). According to one embodiment, a DRL agent is trained in a simulated environment to select QoS settings (e.g., a value of one or more of a minimum IOPS parameter, a maximum IOPS parameter, and a burst IOPS parameter). The training may involve placing the DRL agent into every feasible state representing combinations of QoS settings, workload conditions, and system metrics for a period of time for multiple iterations, and rewarding the DRL agent for selecting QoS settings that minimize an objective function based on a selected measure of system load. The trained DRL agent may then be deployed to one or more DSSs to constantly update QoS settings so as to minimize the selected measure of system load.

A system is described. The system includes a processing resource and a non-transitory computer-readable medium, coupled to the processing resource, having stored therein instructions that when executed by the processing resource cause the processing resource to collect quality of service (QoS) data of a distributed storage system associated with a client device, generate a plurality of QoS diagnostic views to facilitate display of visual representations of the collected QoS data and transmit the visual representations for display at the client device, wherein the data comprises telemetry data, statistics and performance health scores for nodes associated with the client device.

Growing community of inventors

Denver, CO, United States of America

Tyler W Cady