Deepmind Technologies Limited

Marcin Andrychowicz

Joao Ferdinando Gomes De Freitas

Tom Schaul

Sergio Gomez Colmenarejo

Misha Man Ray Denil

Matthew William Hoffman

David Benjamin Pfau

Karol Piotr Kurach

Methods, systems, and apparatus, including computer programs encoded on computer storage media for training machine learning models. One method includes obtaining a machine learning model, wherein the machine learning model comprises one or more model parameters, and the machine learning model is trained using gradient descent techniques to optimize an objective function; determining an update rule for the model parameters using a recurrent neural network (RNN); and applying a determined update rule for a final time step in a sequence of multiple time steps to the model parameters.

Training machine learning models by determining update rules using neural networks

Training machine learning models by determining update rules using recurrent neural networks

Systems, methods, devices, and other techniques are disclosed for using an augmented neural network system to generate a sequence of outputs from a sequence of inputs. An augmented neural network system can include a controller neural network, a hierarchical external memory, and a memory access subsystem. The controller neural network receives a neural network input at each of a series of time steps processes the neural network input to generate a memory key for the time step. The external memory includes a set of memory nodes arranged as a binary tree. To provide an interface between the controller neural network and the external memory, the system includes a memory access subsystem that is configured to, for each of the series of time steps, perform one or more operations to generate a respective output for the time step. The capacity of the neural network system to account for long-range dependencies in input sequences may be extended. Also, memory access efficiency may be increased by structuring the external memory as a binary tree.

Augmenting neural networks with hierarchical external memory

Growing community of inventors

London, United Kingdom

Marcin Andrychowicz