System and method of drainless link repair for increased accelerator utilization during failure

Abstract

A system for repair of an accelerator architecture including a plurality of accelerator host having a plurality of accelerator elements in each host and high-speed interconnects between accelerator elements and including a link health monitor service in the host determining a status of each interconnect internal and external to the host. The link health monitor service detects link status of the interconnects and communicates this to a pod manager responsible for managing a number of interconnected hosts. The pod manager communicates with a scheduler for scheduling processing tasks to each accelerator element or group of elements. Upon detection of an interconnect problem, the pod manager flags a least common ancestor in a binary tree of elements containing elements adjacent to the failed interconnect and flags the ancestor and its parent nodes as unavailable. Pod manager further generates a corrective action repairing the problem interconnect communicating the action to a technician.