Processing real-time processing requests using machine learning models

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for providing consistent processing in a machine learning system are disclosed. A real-time processing request may be received and processed by both a preferred machine learning model and a fallback machine learning model. Processing for the preferred machine learning model may include obtaining additional information. A determination may be made regarding whether the processing of the real-time request by the preferred machine learning model has completed as of an expiration of an acceptable latency period. If the preferred model has not completed as of the expiration of an acceptable latency period, the response to the real-time request may be generated from the fallback model output. If the preferred model has completed prior to or by the expiration of the acceptable latency period, the response to the request may be generated from the preferred model output.