Generating robotic control plans

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for generating a robotic control plan. One of the methods includes obtaining, from a user device, image data depicting an instruction manual for assembling a plurality of assembly components; processing the image data using a machine learning model to generate instruction data representing a sequence of instructions for assembling the plurality of assembly components, wherein the machine learning model has been configured through training to process images depicting instruction manuals and to generate instruction data characterizing sequences of instructions identified in the instruction manuals; processing the instruction data to generate a robotic control plan to be executed by one or more robotic components for assembling the plurality of assembly components; and providing the robotic control plan to a robotic control system for executing the robotic control plan using the one or more robotic components.