On-robot data collection

Abstract

Systems and methods are provided for improved generation and selection of robot sensor data for manual annotation and/or use in training machine learning models used to operate robots. An on-robot controller can operate to determine a cross-modal inconsistency, that a temporally proximate target task was failed, and/or that a confidence in a model output indicate that particular sensor data should be transmitted to a remote system for human annotation and/or use in updating the machine learning model(s) of the robot. Embedding vector(s) representing such selected sensor data (e.g., representing common aspects across a population of sets of sensor data) could also be determined and transmitted to the robot. The robot could then determine embeddings for sensor data and, if the embeddings are similar enough to the transmitted embedding(s), the sensor data could be transmitted to the remote system for annotation and/or model updating.