System and method for allocating resources for heterogeneous service requests

Abstract

A system for allocating constrained resources (e.g., downstream services, execution threads, database connections, input/output channels, computational resources, and/or memory) to requested services that are dependent on those resources may include multiple resource queues, each of which maintains a queue of requests for a respective constrained resource, and multiple service request queues, from which requests may be subsequently serviced. As each request reaches the head of a resource queue, it may receive a resource token for a respective constrained resource. Once the request has collected resource tokens for each of the constrained resources on which it depends, the request may be passed to a service request queue that maintains a queue of requests of a particular type. Requests in the multiple service request queues may be serviced on a round-robin or weighted round-robin basis. The number of tokens available for each constrained resource may be modified based on observed system performance.