System and method for management of inference models of varying complexity

Abstract

Methods and systems for timely execution of inference models hosted by data processing systems are disclosed. To manage inference models hosted by data processing systems, a system may include an inference model manager and any number of data processing systems. The inference models may include inference models with higher complexity topology and inference models with lower complexity topology. The data processing systems may be provided with instructions for when to execute higher complexity topology inference models and when to operate lower complexity topology inference models. The inference model manager may analyze data processing system information and downstream consumer information to determine whether the current deployment of inference models to the data processing systems is capable of meeting the needs of a downstream consumer. Based on the analysis, an execution plan for the inference models and the distribution of inference models may be updated.