Batched reinforcement learning

Abstract

Methods, systems, and apparatus, including computer programs encoded on a computer storage medium, for batched reinforcement learning. For example, the batched reinforcement learning techniques can be used to determine a control policy for a robot in simulation and the control policy can then be used to control the robot in the real world. In one aspect, a method includes obtaining a plurality of current observations, each current observation characterizing a current state of a respective environment replica; processing the current observations in parallel using the action selection neural network in accordance with current values of the network parameters to generate an action batch; obtaining a transition tuple batch including a respective transition tuple for each of the environment replicas, the respective transition tuple for each environment replica including: (i) a subsequent observation and (ii) a reward; and training the action selection neural network on the batch of transition tuples.