Google Inc.

Seagate Technology Incorporated

Gil Shamir

Ara Patapoutian

Zheng Wang

Raman Venkataramani

Lorenzo Coviello

Systems and methods can improve the reproducibility of neural networks by distilling from ensembles. In particular, aspects of the present disclosure are directed to a training scheme that utilizes a combination of an ensemble of neural networks and a single, 'wide' neural network that is more powerful (e.g., exhibits a greater accuracy) than the ensemble. Specifically, the output of the ensemble can be distilled into the single neural network during training of the single neural network. After training, the single neural network can be deployed to generate inferences. In such fashion, the single neural model can provide a superior prediction accuracy while, during training, the ensemble can serve to influence the single neural network to be more reproducible. In addition, an additional single wide tower can be added to generate another output, that can be distilled to the single neural network, to further improve its accuracy.

Distilling from ensembles to improve reproducibility of neural networks

Provided are systems and methods for generating a score for any model which can be updated online, regardless of model type architecture and parameters, leveraging relations between regret and uncertainty.

Forecasting uncertainty in machine learning models

Provided are systems and methods that perform learning to rank using training data for two or more different training lists. Specifically, a training dataset can include a number of training examples. Each training example can include a query and a plurality of items that are potentially responsive to the query. The ranking model can be trained using pairs of items taken from two different training examples.

Cross-list learning to rank

Systems and methods according to the present disclosure can employ a computer-implemented method for inference using a machine-learned model. The method can be implemented by a computing system having one or more computing devices. The method can include obtaining data descriptive of a neural network including one or more network units and one or more gating paths, wherein each of the gating path(s) includes one or more gating units. The method can include obtaining data descriptive of one or more input features. The method can include determining one or more network unit outputs from the network unit(s) based at least in part on the input feature(s). The method can include determining one or more gating values from the gating path(s). The method can include determining one or more gated network unit outputs based at least in part on a combination of the network unit output(s) and the gating value(s).

Minimum deep learning with gating multiplier

Thus, aspects of the present disclosure address model 'blow up' by changing the functionality of the activation, thereby providing 'dead' or &#x201c;dying&#x201d; neurons with the ability to recover from this situation. As one example, for activation functions that have an input region in which the neuron is turned off by a 0 or close to 0 gradient, a training computing system can keep the neuron turned off when the gradient pushes the unit farther into the region (e.g., by applying an update with zero or reduced magnitude). However, if the gradient for the current training example (or batch) attempts to push the unit towards a region in which the neuron is active again, the system can allow for a non-zero gradient (e.g., by applying an update with standard or increased magnitude).

Asymmetric functionality activation for improved stability in neural networks

The present disclosure relates generally to machine learning. More particularly, the present disclosure relates to systems and methods that regularize neural networks by decorrelating neurons or other parameters of the neural networks during training of the neural networks promoting these parameter to innovate over one another.

Systems and methods for regularizing neural networks

Methods, systems, and apparatus, including computer programs encoded on computer storage medium, for regularizing feature weights maintained by a machine learning model. The method includes actions of obtaining a set of training data that includes multiple training feature vectors, and training the machine learning model on each of the training feature vectors, comprising, for each feature vector and for each of a plurality of the features of the feature vector: determining a first loss for the feature vector with the feature, determining a second loss for the feature vector without the feature, and updating a current benefit score for the feature using the first loss and the second loss, wherein the benefit score for the feature is indicative of the usefulness of the feature in generating accurate predicted outcomes for training feature vectors.

Regularization of machine learning models

Technologies are described herein for implementing a universal modulation coding mechanism for a data channel. A user data sequence comprising a first number of bits is received. The user data sequence is encoded into a codeword sequence comprising a second number of bits utilizing a modulation encoder. The modulation encoder may encode the user data sequence based on a set of Markov state transition probabilities. The modulation encoder may implement a data de-compressor algorithm from a lossless data compression scheme. The codeword sequence is output to the data channel. The codeword sequence is subsequently received from the data channel, and the user data sequence is decoded from the codeword sequence utilizing a modulation decoder. The modulation decoder may decode the user data sequence based on the same set of Markov states transition probabilities. The modulation de-coder may implement the data compressor algorithm from the lossless data compression scheme.

Growing community of inventors

Sewickley, PA, United States of America

Gil Shamir