Machine learning model training with hardware error simulation

Abstract

Methods, systems, and apparatus, including computer-readable storage media for handling component error of datacenter infrastructure by training machine learning models to handle errors when the models are deployed. Rather than predict or mitigate errors in the hardware of a datacenter or another site deploying the machine learning model, the model is trained to perform a task with comparable accuracy and efficiency even when some hardware on which the model is deployed fails. Component error can instead be simulated during training to cause the machine learning model deployed on the infrastructure to learn to correct errors caused by the component errors. The model can continue to be trained to compensate for periods in which not all parts of the model are available at inference. Updates to the model can be backpropagated to correct errors for handling instances of component error.