Large-scale retrieval augmented reinforcement learning

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for controlling a reinforcement learning agent in an environment to perform a task. In one aspect, a method comprises: maintaining a retrieval dataset that stores a plurality of history observations and, for each history observation, a respective associated context; receiving a current observation characterizing a current state of the environment; selecting one or more history observations from the plurality of history observations; processing, using an encoder neural network and in accordance with current values of encoder network parameters, an encoder network input comprising (i) the current observation and (ii) the one or more selected history observations and their respective associated context to generate a latent state representation for the current state of the environment; and using the latent state representation to determine an action to be performed by the agent in response to the current observation.