Google Inc.

Sashank Jakkam Reddi

Sanjiv Kumar

Satyen Chandrakant Kale

Manzil Zaheer

Hugh Brendan McMahan

Jakub Konecny

John Keith Rush

Zachary Alan Garrett

Ankit Singh Rawat

Zachary Burr Charles

Aditya Krishna Menon

Generally, the present disclosure is directed to systems and methods that perform adaptive optimization with improved convergence properties. The adaptive optimization techniques described herein are useful in various optimization scenarios, including, for example, training a machine-learned model such as, for example, a neural network. In particular, according to one aspect of the present disclosure, a system implementing the adaptive optimization technique can, over a plurality of iterations, employ an adaptive effective learning rate while also ensuring that the effective learning rate is non-increasing.

Controlled adaptive optimization

A computing system and method can be used to implement a version of federated learning (FL) that incorporates adaptivity (e.g., leverages an adaptive learning rate). In particular, the present disclosure provides a general optimization framework in which (1) clients perform multiple epochs of training using a client optimizer to minimize loss on their local data and (2) a server system updates its global model by applying a gradient-based server optimizer to the average of the clients' model updates. This framework can seamlessly incorporate adaptivity by using adaptive optimizers as client and/or server optimizers. Building upon this general framework, the present disclosure also provides example specific adaptive optimization techniques for FL which use per-coordinate methods as server optimizers. By focusing on adaptive server optimization, the use of adaptive learning rates is enabled without increase in client storage or communication costs and compatibility with cross-device FL can be ensured.

Federated learning with adaptive optimization

Generally, the present disclosure is directed to systems and methods that perform adaptive optimization with improved convergence properties. The adaptive optimization techniques described herein are useful in various optimization scenarios, including, for example, training a machine-learned model such as, for example, a neural network. In particular, according to one aspect of the present disclosure, a system implementing the adaptive optimization technique can, over a plurality of iterations, employ an adaptive learning rate while also ensuring that the learning rate is non-increasing.

Adaptive optimization with improved convergence

A method for training a machine learning model, e.g., a neural network, using a regularization scheme is disclosed. The method includes generating regularized partial gradients of losses computed using an objective function for training the machine learning model.

Training machine learning models to be robust against label noise

Growing community of inventors

Jersey City, NJ, United States of America

Sashank Jakkam Reddi