Reinforcement learning with centralized inference and training

Abstract

Methods, systems, and apparatus, including computer programs encoded on computer storage media, for performing reinforcement learning with centralized inference and training One of the methods includes receiving, at a current time-step in a plurality of time-steps, a respective observation by an actor for each environment of a plurality of environments; obtaining, for each environment, a respective reward for the actor as a result of the actor performing a respective action at a previous time-step preceding the current time-step; processing, for each environment, the respective observation and respective reward through a policy model; providing, to the actor, the respective policy outputs for each of the plurality of environments; maintaining at a repository and for each environment, a respective sequence of tuples corresponding to the actor; determining that a maintained sequence meets a threshold condition; and in response, training the policy model on the maintained sequence.