Microsoft Technology Licensing, LLC

Qiwei Ye

Tie-Yan Liu

Wei Ting Chen

Wei Chen

Fei Gao

Taifeng Wang

Qi Meng

Various implementations relate to asynchronous training of a machine learning model. A server receives feedback data generated by training the machine learning model from a worker. The feedback data are obtained by the worker with its own training data and are associated with previous values of a set of parameters of the machine learning model at the worker. The server determines differences between the previous values and current values of the set of parameters at the server. The current value may have been updated for once or more due to operation of other workers. Then, the server can update the current values of the set of parameters based on the feedback data and the differences between values of the set of parameters. Thus, the updating does not only take the training result of each worker into consideration but also makes proper compensation for delay between different workers.

Asychronous training of machine learning model

In implementations of the present disclosure, a solution for optimization of a learning network in an equivalent class space is provided. In this solution, base paths running through layers of a learning network are determined. Each node utilizes an activation function with a scaling invariant property to process an input from a node of a previous layer, each base path comprises a single node in each layer, and processing in the base paths is linearly independent from each other. A combined value of parameters associated with nodes in each base path is updated. A parameter associated with a node is used to adjust an input obtained from a node of a previous layer. Values of parameters associated with nodes in the base paths are updated based on updated combined values of parameters. Through this solution, optimization efficiency can be improved and more accurate optimized values of parameters are achieved.

Optimization of neural network in equivalent class space

Growing community of inventors

Redmond, WA, United States of America

Qiwei Ye