Bridging language and environments with rendering functions and vision-language models

Abstract

A robot system includes: image encodings generated based on renderings of configurations, respectively of a robot, the configurations including at least a predetermined number of different poses of the robot in an environment; an encoding module configured to receive text descriptive of an action to be performed by the robot and to encode the text into a text encoding; a scoring module configured to generate scores for the configurations based on comparisons of (a) the text encoding with (b) the image encoding of the respective configuration; a selection module configured to select k of the configurations based on the scores, where k is an integer greater than or equal to 1; and an actuation module configured to actuate the robot based on the selected k of the configurations based on actuating the robot to achieve the action described in the text.