Edge domain-specific accelerator virtualization and scheduling

Abstract

Presented herein are embodiments to implement a temporal queueing system with class-based fair queuing and dynamic resource allocation based on a novel look-ahead capability to manage various models and workloads for utilization/efficiency improvements. Embodiments may be implemented to allocate accelerator resources based on platform-defined timeslots, and therefore significantly increase the ability of workloads to access hardware accelerator resources. Training and inference may be supported with flexible preemption and the ability to support run-to-completion for training tasks while still supporting non-run-to-completion for inference tasks. Embodiments may be implemented by an edge software operation platform through virtual accelerators to allow emulation of different types of hardware accelerators and to map to the hardware accelerators with hardware-specific procedures managed by an edge orchestrator and an edge endpoint. Accordingly, embodiments of the present disclosure reduce the requirements for the workload to manage platform capacity and hardware.