Performing inference using an adaptive, hybrid local/remote technique

Abstract

A hybrid inference facility receives a sequence of data items. For each data item, the facility: forwards the data item to a server; subjects it to a local machine learning model to produce a local inference result for the data item; and the local inference result to a queue; aggregates the inference results contained by the queue to obtain an output inference result; and removes the oldest inference result from the queue. The facility receives from the server cloud inference results each obtained by applying a server machine learning model to one of the data items forwarded to the server. For each received cloud inference result, the facility substitutes the cloud inference result in the queue for the local inference result for the same data item.