Optimum cessation of training in neural networks

Abstract

A system and method for training a neural network that ceases training at or near the optimally trained point is presented. A neural network having an input layer, a hidden layer, and an output layer with each layer having one or more nodes is presented. Each node in the input layer is connected to each node in the hidden layer and each node in the hidden layer is connected to each node in the output layer. Each connection between nodes has an associated weight. All nodes in the input layer are connected to a different historical datum from the set of historical data. The neural network being operative by outputting a prediction or classification, the output of the output layer nodes, when presented with input data. The weights associated with the connections of the neural network are first adjusted by a training device. The training device then iteratively applies a training set to the neural network, the training set consisting of historical data. After each iteration the weights associated with the connections are adjusted according to the difference between the prediction or classification produced by the neural network given the training data and the known prediction or classification of the historical data. Additionally, after each iteration, a test set, consisting of different historical data from that in the training set, is presented to the neural network. The training device then determines the difference between the known result from the test set and the result from the presentation of the test set to the neural network. This difference, herein referred to as the variance, is then recorded along with the weights in the neural network. The variance is monitored at each iteration to determine if it is monotonically, within a given margin of error, decreasing. That is the prediction or classification resulting from the test set being presented to the neural network is getting successively closer to matching the known result from the test set. When the variance hits the inflection point where it begins to increase, training is ceased. At this point the neural network is no longer learning the pattern underlying the input data, but is instead over fitting the input data.