Hybrid queue system for request throttling

Abstract

Systems and related methods are disclosed to store and throttle requests received by a service provider. In embodiments, the system includes two queues, a first-in-first-out (FIFO) queue and an overflow queue. An incoming request is stored in the overflow queue when there is no room in the FIFO queue. The overflow queue stores the requests in some priority order, which determines the order that the requests are promoted onto the FIFO queue and throttled. The FIFO queue may be sized according to a response time requirement provided in a service level agreement (SLA). In some embodiments, the FIFO queue may dynamically adjust its size based on the expected processing time or abandon duration of incoming requests. The hybrid approach allows a system to handle requests in simple FIFO order in normal circumstances, and in a more sophisticated priority order when the system is overloaded.