Event-based task processing system and method based on a stateful distributed streaming queue

Abstract

Various methods, apparatuses/systems, and media for event-based task processing are disclosed. The system includes a processor and a memory operatively connected to the processor via a communication interface. The processor defines a stateful distributed stream queue as a streaming queue which holds stateful computational requests generated at a stream of events, wherein each request contains a context and a collection of queue tasks processed by workers dedicated to the queue; runs the stateful distributed stream queue on a cloud infrastructure; implements the stateful computational requests, by utilizing the stateful distributed stream queue, at the stream of events; and clusters the queue tasks into affinity sub-queues by implementing configurable classification criteria to indicate that their computation is sped up on particular workers, while workers process tasks specific to their affinity; and implements abstractions for submitting and automatically re-computing the queue tasks while maintaining queue state between streaming events.