Dispatch engine processing multiple task types for efficient dispatch of execution of transformer workloads on ai compute accelerator

Abstract

An AI accelerator comprises a plurality of chiplets and a dispatch engine integrated circuit, is configurable to efficiently process AI computational workloads (e.g., transformer operations). The dispatch engine includes a task fetch engine and a plurality of task pipelines coupled thereto. Each task pipeline comprises a task queue, tasks coupled to the task queue, and a task dispatch engine coupled to the tasks and configured to dispatch one or more of the tasks. Three task types are used: instruction tasks, wait tasks, and completion tasks. An instruction dispatch engine coupled to the task dispatch engine is configured to receive and dispatch an instruction task to one or more compute units. A memory device coupled to the instruction dispatch engine stores one or more instructions. An instruction fetch engine is coupled to the memory device and configured to transfer one or more instructions from an external memory to the memory device.