8
$\begingroup$

I'm a bit troubled and confused by the idea of how the technique early stopping is defined. If you take a look it Wikipedia, it is defined as follows:

  1. Split the training data into a training set and a validation set, e.g. in a 2-to-1 proportion.
  2. Train only on the training set and evaluate the per-example error on the validation set once in a while, e.g. after every fifth epoch.
  3. Stop training as soon as the error on the validation set is higher than it was the last time it was checked.
  4. Use the weights the network had in that previous step as the result of the training run.

I was using the method myself in my experiments (with 10-fold cross-validation). I'm checking the validation error on each epoch (and also calculate the validation accuracy) and set a patience parameter of 2. That means, if the validation error increases for 2 epochs in a row -> stop training. Then I used the results of the last epoch when the model finished.

Ian Goodfellow uses another definition in his deep learning book. As 4th step he suggests using the weights of the best working model (i.e. save the model each time the validation error is checked).

I don't need the saved model, I only need the results for my work. So for me the proposed early stopping by Goodfellow would mean I'd just take the highest validation accuracy I've reached for my final result? Somehow this doesn't seem legit. I don't have this information in a real-world situation when there is no development set. But in that case, what is the reason to use early stopping in the first place? Determining the number of epochs by e.g. averaging the number of epochs for the folds and use it for the test run later on?

$\endgroup$
  • $\begingroup$ Usually you would have separate test data that you would use to get an unbiased estimate of your model performance. $\endgroup$ – Aaron Sep 28 '17 at 20:18
  • $\begingroup$ Yes. I do have training, development and test split. But using early stopping on the test split would be cheating. So I can't use the early stopping method on the test set for the final run. In that case early stopping would only be useful to figure out how many epochs to run on the test set (in order to prevent overfitting). As far as I understand it by now. $\endgroup$ – V1nc3nt Sep 28 '17 at 20:43
  • $\begingroup$ It should be pointed out that a similar question exists, but as this question here is about a single model, while the earlier question refers to some kind of a mixture of models (or so I believe - to be honest, both the question and answer there are rather difficult to understand for me). So I do not feel that the earlier question has answers to the questions raised by OP here. $\endgroup$ – fnl Sep 29 '17 at 12:28
7
$\begingroup$

Determining the number of epochs by e.g. averaging the number of epochs for the folds and use it for the test run later on?

Shortest possible answer: Yes! But let me add some context...

I believe you are referring to Section 7.8, pages 246ff, on Early Stopping in the Deep Learning book. The described procedure there, however, is significantly different from yours. Goodfellow et al. suggest to split your data in three sets first: a training, dev, and test set. Then, you train (on the training set) until the error from that model increases (on the dev set), at which point you stop. Finally, you use the trained model that had the lowest dev set error and evaluate it on the test set. No cross-validation involved at all.

However, you seem to be trying to do both early stopping (ES) and cross-validation (CV), as well as model evaluation all on the same set. That is, you seem to be using all your data for CV, training on each split with ES, and then using the average performance over those CV splits as your final evaluation results. If that is the case, that indeed is stark over-fitting (and certainly not what is described by Goodfellow et al.), and your approach gives you exactly the opposite result of what ES is meant for -- as a regularization technique to prevent over-fitting. If it is not clear why: Because you've "peaked" at your final evaluation instances during training time to figure out when to ("early") stop training; That is, you are optimizing against the evaluation instances during training, which is (over-) fitting your model (on that evaluation data), by definition.

So by now, I hope to have answered your other [two] questions.

The answer by the higgs broson (to your last question, as cited above) already gives a meaningful way to combine CV and ES to save you some training time: You could split your full data in two sets only - a dev and a test set - and use the dev set to do CV while applying ES on each split. That is, you train on each split of your dev set, and stop once the lowest error on the training instances you set aside for evaluating that split has been reached [1]. Then you average the number of epochs needed to reach that lowest error from each split and train on the full dev set for that (averaged) number of epochs. Finally, you validate that outcome on the test set you set aside and haven't touched yet.

[1] Though unlike the higgs broson I would recommend to evaluate after every epoch. Two reasons for that: (1), comparative to training, the evaluation time will be negligible. (2), imagine your min. error is at epoch 51, but you evaluate at epoch 50 and 60. It isn't unlikely that the error at epoch 60 will be lower than at epoch 50; Yet, you would choose 60 as your epoch parameter, which clearly is sub-optimal and in fact even going a bit against the purpose of using ES in the first place.

$\endgroup$
  • $\begingroup$ Thank you for your answer. I already split my data in dev and test. I never touched the test set. I further split the dev set in 10 folds and do the CV on it. 9 folds training, 1 fold dev. The dev fold is used for ES and checked each epoch already. Since I didn't elaborate how I do my split (sorry!); am I already doing what you suggested me to do or did I get you wrong somewhere? $\endgroup$ – V1nc3nt Sep 29 '17 at 13:15
  • $\begingroup$ Well done - you're already doing everything correctly, I'd say! $\endgroup$ – fnl Sep 29 '17 at 17:58
  • $\begingroup$ So in the end early stopping is just a way to tune the hyper-parameter "number of epochs". I started using it under the wrong impression and I was afraid I had to start over again and repeat all my tests, because I used it in a wrong way. But I can smoothly incorporate it. Thanks for your help. $\endgroup$ – V1nc3nt Sep 30 '17 at 10:21
3
$\begingroup$

The way that you can use cross-validation to determine the optimal number of epochs to train with early stopping is this: suppose we were training for between 1 to 100 epochs. For each fold, train your model and record the validation error every, say, 10 epochs. Save these trajectories of validation error vs number of epochs trained and average them together over all folds. This will yield an "average test error vs epoch" curve. The stopping point to use is the number of epochs that minimizes the average test error. You can then train your network on the full training set (no cross validation) for that many epochs.

The purpose of early stopping is to avoid overfitting. You use N-fold cross-validation to estimate the generalization error of your model by creating N synthetic train/test sets and (usually) averaging together the results. Hopefully, the test set (aka new real-world data) that you are given later is going to be similar enough to the synethetic test sets that you generated with CV so that the stopping point you found earlier is close to optimal given this new testing data.

$\endgroup$
  • $\begingroup$ What you describe in your first paragraph isn't really the "early stopping" as proposed I think, since you'd have to run your model for these 100 epochs on each fold to compare the results, right? But usually you want to stop early on the cross-validation as well, if I understand correctly (saves a lot of time, too). Anyway, it seems to be a reasonable way to figure out how many epochs to run for the final test run. But wouldn't you rather use early stopping as proposed (also for the CV) and use the method you describe in your first paragraph on the best, final tuned model? $\endgroup$ – V1nc3nt Sep 28 '17 at 20:10
  • 1
    $\begingroup$ How I've described "early stopping" is how (I believe) most people would do it. Usually, when one is generating test error vs epoch curves in CV, you just fix the total number of epochs you train for. Hopefully by the time you've reached the max number of allowed epochs the error curve has flattened out. If not, go back and repeat the process with a larger max epoch limit. On the other hand, if your program tells you that the test error has plateaued by 50 epochs, then you don't need to train it out to 100. The goal is to just train enough so that you've seen your test error stop improving. $\endgroup$ – the higgs broson Sep 28 '17 at 20:19
  • 1
    $\begingroup$ I also want to point out that another reason for keeping the max training epoch number consistent across folds is that while in 1 fold the test error may stop improving at 60 epochs, but in another it may stop at 80 epochs. When you average all of the test error curves together across folds to determine the real stopping point, you’ll need test error data at each epoch for each fold. Stopping the training within a fold early may prevent you from having that. $\endgroup$ – the higgs broson Sep 28 '17 at 20:36
  • $\begingroup$ Actually I first started using early stopping, because training takes ages. The model has a lot of parameters and data. So I wanted it to stop as soon as the validation error increases (for 2 epochs in a row as so-called "patience" parameter). So I can't really train it until learning stops. But still I can do what you suggest on the final model then and compare the error trajectories from the epochs of all its folds and take the number of epoch which has the lowest averaged error. Did I understand you correctly? Of course I can only compare up to the minimum of computed epochs of all folds. $\endgroup$ – V1nc3nt Sep 28 '17 at 20:39
  • 1
    $\begingroup$ Vincent, I think you understand correctly. When averaging the error curves together you must indeed stop at the minimum epoch number across folds. To be safe, I’d recommend going back and training the models in other folds up to the maximum stopping epoch found. $\endgroup$ – the higgs broson Sep 28 '17 at 20:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.