Training a simulation-to-real model, based on perception losses, to mitigate reality gap in robotics and/or other applications

Abstract

Implementations disclosed herein relate to mitigating the reality gap through training a simulation-to-real machine learning model ('Sim2Real' model) based on loss(es) generated using already trained perception model(s), such as perception model(s) trained to process an image and generate predicted output(s) (e.g., predicted bounding area(s) and/or predicted classification(s) for object(s) in a given image). In some implementations, the perception model(s) can be trained to generate predicted output(s) based on processing real image(s), and can be used to process a predicted real image to generate the predicted output(s). The predicted output(s) can be compared to ground truth output(s), from the simulator, to generate a perception loss. The Sim2Real model can be updated based on the generated perception loss. In various implementations, the Sim2Real model can additionally and/or alternatively be updated based on additional loss(es). Once updated, the Sim2Real model can be utilized in generating robotic control policies to control a real robot.