Microsoft Technology Licensing, LLC

Ge Yang

Nicolo Fusi

Francesco Paolo Casale

Saeed Amizadeh

Douglas Christopher Burger

Jianfeng Gao

Eric S Chung

Bita Darvish Rouhani

Ritchie Zhao

Xiaodong Liu

Maximilian Golub

Jingfeng Hu

Systems and method are provided that are directed to tuning a hyperparameter associated with a small neural network model and transferring the hyperparameter to a large neural network model. At least one neural network model may be received along with a request for one or more tuned hyperparameters. Prior to scaling the large neural network, the large neural network is parameterized in accordance with a parameterizing scheme. The large neural network is then scaled and reduced in size such that a hyperparameter tuning process may be performed. A tuned hyperparameter may then be provided to a requestor such that the hyperparameter can be directly input into the large neural network. By tuning a hyper parameter using a small neural network, significant computation cycles and energy may be saved.

Hyperparameter transfer via the theory of infinite-width neural networks

Systems and methods, for selecting a neural network for a machine learning (ML) problem, are disclosed. A method includes accessing an input matrix, and accessing an ML problem space associated with an ML problem and multiple untrained candidate neural networks for solving the ML problem. The method includes computing, for each untrained candidate neural network, at least one expressivity measure capturing an expressivity of the candidate neural network with respect to the ML problem. The method includes computing, for each untrained candidate neural network, at least one trainability measure capturing a trainability of the candidate neural network with respect to the ML problem. The method includes selecting, based on the at least one expressivity measure and the at least one trainability measure, at least one candidate neural network for solving the ML problem. The method includes providing an output representing the selected at least one candidate neural network.

Selecting a neural network architecture for a supervised machine learning problem

Embodiments of the present disclosure include a system for optimizing an artificial neural network by configuring a model, based on a plurality of training parameters, to execute a training process, monitoring a plurality of statistics produced upon execution of the training process, and adjusting one or more of the training parameters, based on one or more of the statistics, to maintain at least one of the statistics within a predetermined range. In some embodiments, artificial intelligence (AI) processors may execute a training process on a model, the training process having an associated set of training parameters. Execution of the training process may produce a plurality of statistics. Control processor(s) coupled to the AI processor(s) may receive the statistics, and in accordance therewith, adjust one or more of the training parameters to maintain at least one of the statistics within a predetermined range during execution of the training process.

System for training an artificial neural network

Systems and methods for selecting a neural network for a machine learning problem are disclosed. A method includes accessing an input matrix. The method includes accessing a machine learning problem space associated with a machine learning problem and multiple untrained candidate neural networks for solving the machine learning problem. The method includes computing, for each untrained candidate neural network, at least one expressivity measure capturing an expressivity of the candidate neural network with respect to the machine learning problem. The method includes computing, for each untrained candidate neural network, at least one trainability measure capturing a trainability of the candidate neural network with respect to the machine learning problem. The method includes selecting, based on the at least one expressivity measure and the at least one trainability measure, at least one candidate neural network for solving the machine learning problem. The method includes providing an output representing the selected at least one candidate neural network.

Growing community of inventors

Bellevue, WA, United States of America

Ge Yang