Shadow testing of contextual machine learning models

Abstract

Shadow testing may be performed to determine whether an updated machine learning model (e.g., an updated machine learning algorithm) is suitable for deployment and replacement of a currently deployed machine learning model. In certain instances, the shadow testing involves machine learning models that determine predictions on the context of language-based user interactions (e.g., the machine learning algorithms are contextual machine learning algorithms). The shadow testing may include asynchronous determinations of predictions by the updated machine learning model on live traffic data being provided to the existing machine learning model. From the shadow testing, disagreement rates between the models and accuracy rates of the models may be used to determine whether the updated machine learning model should be deployed.