Understanding early stopping in neural networks and its implications when using cross-validation

Question

I'm a bit troubled and confused by the idea of how the technique early stopping is defined. If you take a look it Wikipedia, it is defined as follows:

1. Split the training data into a training set and a validation set, e.g. in a 2-to-1 proportion.
2. Train only on the training set and evaluate the per-example error on the validation set once in a while, e.g. after every fifth epoch.
3. Stop training as soon as the error on the validation set is higher than it was the last time it was checked.
4. Use the weights the network had in that previous step as the result of the training run.

I was using the method myself in my experiments (with 10-fold cross-validation). I'm checking the validation error on each epoch (and also calculate the validation accuracy) and set a patience parameter of 2. That means, if the validation error increases for 2 epochs in a row -> stop training. Then I used the results of the last epoch when the model finished.

Ian Goodfellow uses another definition in his deep learning book. As 4th step he suggests using the weights of the best working model (i.e. save the model each time the validation error is checked).

I don't need the saved model, I only need the results for my work. So for me the proposed early stopping by Goodfellow would mean I'd just take the highest validation accuracy I've reached for my final result? Somehow this doesn't seem legit. I don't have this information in a real-world situation when there is no development set. But in that case, what is the reason to use early stopping in the first place? Determining the number of epochs by e.g. averaging the number of epochs for the folds and use it for the test run later on?

Answer 1

Determining the number of epochs by e.g. averaging the number of epochs for the folds and use it for the test run later on?

Shortest possible answer: Yes! But let me add some context...

I believe you are referring to Section 7.8, pages 246ff, on Early Stopping in the Deep Learning book. The described procedure there, however, is significantly different from yours. Goodfellow et al. suggest to split your data in three sets first: a training, dev, and test set. Then, you train (on the training set) until the error from that model increases (on the dev set), at which point you stop. Finally, you use the trained model that had the lowest dev set error and evaluate it on the test set. No cross-validation involved at all.

However, you seem to be trying to do both early stopping (ES) and cross-validation (CV), as well as model evaluation all on the same set. That is, you seem to be using all your data for CV, training on each split with ES, and then using the average performance over those CV splits as your final evaluation results. If that is the case, that indeed is stark over-fitting (and certainly not what is described by Goodfellow et al.), and your approach gives you exactly the opposite result of what ES is meant for -- as a regularization technique to prevent over-fitting. If it is not clear why: Because you've "peaked" at your final evaluation instances during training time to figure out when to ("early") stop training; That is, you are optimizing against the evaluation instances during training, which is (over-) fitting your model (on that evaluation data), by definition.

So by now, I hope to have answered your other [two] questions.

The answer by the higgs broson (to your last question, as cited above) already gives a meaningful way to combine CV and ES to save you some training time: You could split your full data in two sets only - a dev and a test set - and use the dev set to do CV while applying ES on each split. That is, you train on each split of your dev set, and stop once the lowest error on the training instances you set aside for evaluating that split has been reached [1]. Then you average the number of epochs needed to reach that lowest error from each split and train on the full dev set for that (averaged) number of epochs. Finally, you validate that outcome on the test set you set aside and haven't touched yet.

[1] Though unlike the higgs broson I would recommend to evaluate after every epoch. Two reasons for that: (1), comparative to training, the evaluation time will be negligible. (2), imagine your min. error is at epoch 51, but you evaluate at epoch 50 and 60. It isn't unlikely that the error at epoch 60 will be lower than at epoch 50; Yet, you would choose 60 as your epoch parameter, which clearly is sub-optimal and in fact even going a bit against the purpose of using ES in the first place.

Answer 2

The way that you can use cross-validation to determine the optimal number of epochs to train with early stopping is this: suppose we were training for between 1 to 100 epochs. For each fold, train your model and record the validation error every, say, 10 epochs. Save these trajectories of validation error vs number of epochs trained and average them together over all folds. This will yield an "average test error vs epoch" curve. The stopping point to use is the number of epochs that minimizes the average test error. You can then train your network on the full training set (no cross validation) for that many epochs.

The purpose of early stopping is to avoid overfitting. You use N-fold cross-validation to estimate the generalization error of your model by creating N synthetic train/test sets and (usually) averaging together the results. Hopefully, the test set (aka new real-world data) that you are given later is going to be similar enough to the synethetic test sets that you generated with CV so that the stopping point you found earlier is close to optimal given this new testing data.