Dynamic cross protocol tuner

Abstract

Implementations are provided herein for dynamically tuning resource allocation within a node among a cluster of nodes of a distributed file system. Each node can have a plurality of protocol heads that enable the node to communicate with clients using varying protocols. As tasks are received by the node from the client through the protocol head, the amount of work required to perform the task can be estimated, the amount of resources consumed by the task can be estimated, the amount of kernel memory required to perform the task can be estimated, and the amount of general memory required to perform the task can be estimated. Using these estimations, tasks can be scheduled by a scheduler based on the most efficient means to concurrently process as many tasks as possible using available resources of the node. In addition to efficiently scheduling tasks by the scheduler, a dynamic tuner can also reside on the node that can monitor all work executing on the node for each protocol head. The tuner can then distribute and, over time, redistribute resources to each protocol head using real-time data to make adjustments. In addition to tuning resources allocated to the protocol heads, the tuner can also account for pending and scheduled system task activity, as well as account for misbehaving clients.