As we train a neural network, we have access to the error-rate (both on training, and test patterns). What are standard techniques to use this information to stop the learning as quickly as possible when over-fitting starts?
Tell me more
×
Cross Validated is a question and answer site for
statisticians, data analysts, data miners and data visualization experts. It's 100% free, no registration required.
|
|
Some rather disorganized thoughts on this issue (I hope there is something of use in there somewhere): Rather than having training and test data, you ought to have three partitions: (i) the training set, which is used to optimize the weights of the network (ii) a validation set, which is used to decide when to stop training (and to make other choices about the model such as the number of hidden layer neurons to use) and (iii) the test set, which is used to estimate the performance of the final network. You need three partitions rather than two in order to get an unbiased performance estimate. As an aspect of the model has been tuned to maximize performance on both the training set and the validation set (via the choice of when to stop training etc), which means that the performance on both of those sets will give an optimistically biased estimate of true generalization performance (probably rather strongly biased). The basic idea of early stopping is based on the assumption that initially the weights of the network will be changed in ways that learn the underlying structure of the data, but after some time genuine improvements in generalization will no longer be available. When the network gets to that point it can often still reduce the error on the training set by memorizing the noise in the data, which generally results in generalization performance becoming worse. However if we monitor the performance on a separate set of data, we should see the error on that set start to rise once we move from the first phase of learning (which is beneficial) to the second (which isn't). The simplest thing to do is simply to monitor the validation set performance and save a copy of the network every time we see a validation error that is lower than the best we have seen so far, and then simply use that network to make predictions. The problem is that the validation set performance is often rather noisy, so it is difficult to know whether we are likely to see a better network if we continue training, or whether the improvements in the validation set are meaningful. I generally used to just train to convergence and keep the set of weights that minimized the validation set error along the way. There is a good book called "Neural Networks: Tricks of the Trade", edited by Genevieve Orr and Klass-Robert Muller, which is a collection of advice from many leading neural network experts of the 1990s. At least one of these gives some sensible advice on early stopping. These days I prefer regularization instead, Chris Bishops excellent book "neural networks for pattern recognition" uses this approach and explains its relationship to early stopping. It is far easier in practice in my experience, although more modern approaches, such as kernel methods or Gaussian processes tend to be better still. |
|||
|
|
|
There is no standard method of doing it because it is wrong -- you simply overfit your model on the testing data (it is a hidden form of overfitting by parameter selection). |
|||||||||||||
|
