The State University of New York

Lei Yu

Robert Wright

Steven Loscalzo

Andrew Cohen

A system and method for controlling a system, comprising estimating an optimal control policy for the system; receiving data representing sequential states and associated trajectories of the system, comprising off-policy states and associated off-policy trajectories; improving the estimate of the optimal control policy by performing at least one approximate value iteration, comprising: estimating a value of operation of the system dependent on the estimated optimal control policy; using a complex return of the received data, biased by the off-policy states, to determine a bound dependent on at least the off-policy trajectories, and using the bound to improve the estimate of the value of operation of the system according to the estimated optimal control policy; and updating the estimate of the optimimal control policy, dependent on the improved estimate of the value of operation of the system. The control system may produce an output signal to control the system directly, or output the optimized control policy. The system preferably is a reinforcement learning system which continually improves.

Approximate value iteration with complex returns by bounding

The present technology addresses the problem of quickly and safely improving policies in online reinforcement learning domains. As its solution, an exploration strategy comprising diverse exploration (DE) is employed, which learns and deploys a diverse set of safe policies to explore the environment. DE theory explains why diversity in behavior policies enables effective exploration without sacrificing exploitation. An empirical study shows that an online policy improvement algorithm framework implementing the DE strategy can achieve both fast policy improvement and safe online performance.

Framework and methods of diverse exploration for fast and safe policy improvement

A control system and method for controlling a system, which employs a data set representing a plurality of states and associated trajectories of an environment of the system; and which iteratively determines an estimate of an optimal control policy for the system. The iterative process performs the substeps, until convergence, of estimating a long term value for operation at a respective state of the environment over a series of predicted future environmental states; using a complex return of the data set to determine a bound to improve the estimated long term value; and producing an updated estimate of an optimal control policy dependent on the improved estimate of the long term value. The control system may produce an output signal to control the system directly, or output the optimized control policy. The system preferably is a reinforcement learning system which continually improves.

Growing community of inventors

Vestal, NY, United States of America

Lei Yu