System and method for executing multiple inference models using inference model prioritization

Abstract

Methods and systems for managing execution of inference models across multiple data processing systems are disclosed. To manage execution of inference models across multiple data processing systems, a system may include an inference model manager and any number of data processing systems. The inference model manager may obtain operational capability data for the inference models from the data processing systems. The inference model manager may use the operational capability data to determine whether the data processing systems have access to sufficient computing resources to complete timely execution of the inference models. If the data processing systems do not have access to sufficient computing resources to complete timely execution of the inference models, the inference model manager may re-assign one or more data processing systems to re-balance the computing resource load and support continued operation of at least a portion of the inference models.