As we train a neural network, we have access to the error-rate (both on training, and test patterns). What are standard techniques to use this information to stop the learning as quickly as possible when over-fitting starts?
$\begingroup$
$\endgroup$
5
asked Jun 5, 2012 at 0:35
-
1$\begingroup$ This is known as "early stopping". Maybe this helps you a little bit: en.wikipedia.org/wiki/Early_stopping . $\endgroup$– alfaCommented Jun 6, 2012 at 13:22
-
$\begingroup$ @alfa thank you for the link, the name is helpful. Unfortunately the task I am trying to solve is specifically in the issues section: "Notion of "increasing validation error" is ambiguous; it may go up and down numerous times during training." $\endgroup$– Artem KaznatcheevCommented Jun 6, 2012 at 16:46
-
1$\begingroup$ Yes, I know. :D I would like to hear an answer, too. But I think there is no solution. You can save the best results and train until the algorithm converges or you could try Bayesian Neural Networks. $\endgroup$– alfaCommented Jun 7, 2012 at 13:20
-
2$\begingroup$ I noticed your request for migrating to CS.SE. Please, confirm whether you mean cs.stackexchange.com or scicomp.stackexchange.com. This question is largely on-topic on this site, though. Maybe you can update it to bump it up and attract more attention, or offer a bounty. $\endgroup$– chlCommented Aug 16, 2012 at 10:21
-
$\begingroup$ @alfa - good comments (+1). While Bayesian neural nets are much more convenient, they can also go badly wrong if the model is mis-specified, so in practice they can perform rather poorly. I have been performing an empirical study of this sort of thing, and am just writing up the paper. $\endgroup$– Dikran MarsupialCommented Aug 16, 2012 at 13:44
Add a comment
|
2 Answers
$\begingroup$
$\endgroup$
Some rather disorganized thoughts on this issue (I hope there is something of use in there somewhere):
Rather than having training and test data, you ought to have three partitions: (i) the training set, which is used to optimize the weights of the network (ii) a validation set, which is used to decide when to stop training (and to make other choices about the model such as the number of hidden layer neurons to use) and (iii) the test set, which is used to estimate the performance of the final network. You need three partitions rather than two in order to get an unbiased performance estimate. As an aspect of the model has been tuned to maximize performance on both the training set and the validation set (via the choice of when to stop training etc), which means that the performance on both of those sets will give an optimistically biased estimate of true generalization performance (probably rather strongly biased).
The basic idea of early stopping is based on the assumption that initially the weights of the network will be changed in ways that learn the underlying structure of the data, but after some time genuine improvements in generalization will no longer be available. When the network gets to that point it can often still reduce the error on the training set by memorizing the noise in the data, which generally results in generalization performance becoming worse. However if we monitor the performance on a separate set of data, we should see the error on that set start to rise once we move from the first phase of learning (which is beneficial) to the second (which isn't). The simplest thing to do is simply to monitor the validation set performance and save a copy of the network every time we see a validation error that is lower than the best we have seen so far, and then simply use that network to make predictions.
The problem is that the validation set performance is often rather noisy, so it is difficult to know whether we are likely to see a better network if we continue training, or whether the improvements in the validation set are meaningful. I generally used to just train to convergence and keep the set of weights that minimized the validation set error along the way.
There is a good book called "Neural Networks: Tricks of the Trade", edited by Genevieve Orr and Klass-Robert Muller, which is a collection of advice from many leading neural network experts of the 1990s. At least one of these gives some sensible advice on early stopping.
These days I prefer regularization instead, Chris Bishops excellent book "neural networks for pattern recognition" uses this approach and explains its relationship to early stopping. It is far easier in practice in my experience, although more modern approaches, such as kernel methods or Gaussian processes tend to be better still.
answered Aug 16, 2012 at 13:42
$\begingroup$
$\endgroup$
5
There is no standard method of doing it because it is wrong -- you simply overfit your model on the testing data (it is a hidden form of overfitting by parameter selection).
-
$\begingroup$ Care to elaborate why it is wrong? Or references? $\endgroup$– bayerjCommented Aug 16, 2012 at 18:48
-
$\begingroup$ @bayerj Accuracy on a testing set is reliable only if strictly no information is used to build a model. In this case, you explicitly parametrise model to achieve best accuracy on this "test" set, which is thus no longer eligible for a reliable assessment of the true model performance. $\endgroup$– user88Commented Aug 16, 2012 at 19:40
-
1$\begingroup$ Which is why it is common practice to use a training, validation and testing set: training for optimization of the loss, validation for telling when to stop optimization. Evaluation of the model (in the very end) on the test set. $\endgroup$– bayerjCommented Aug 17, 2012 at 8:07
-
$\begingroup$ @bayerj Note that OP stated no intention of using validation set, thus my answer. Anyway, even with a proper evaluation you still don't know whether the fitted parameter is optimal or overfitted because you have no datum. <rant>The fundamental problem here is that ANNs have no built-in means of leveraging parsimony and fitness, thus it is almost impossible to train them in a robust way and thus reasonable people have dropped this technology ages ago.</rant> $\endgroup$– user88Commented Aug 17, 2012 at 14:17
-
$\begingroup$ I was just pointing to the way to do it right. <rant>And in the meantime, unreasonable people have stopped worrying about optimality and build cool stuff that works and delivers consumer satisfaction with: neural networks.</rant> $\endgroup$– bayerjCommented Aug 21, 2012 at 19:21