I'm a bit troubled and confused by the idea of how the technique early stopping is defined. If you take a look it Wikipedia, it is defined as follows:
- Split the training data into a training set and a validation set, e.g. in a 2-to-1 proportion.
- Train only on the training set and evaluate the per-example error on the validation set once in a while, e.g. after every fifth epoch.
- Stop training as soon as the error on the validation set is higher than it was the last time it was checked.
- Use the weights the network had in that previous step as the result of the training run.
I was using the method myself in my experiments (with 10-fold cross-validation). I'm checking the validation error on each epoch (and also calculate the validation accuracy) and set a patience parameter of 2. That means, if the validation error increases for 2 epochs in a row -> stop training. Then I used the results of the last epoch when the model finished.
Ian Goodfellow uses another definition in his deep learning book. As 4th step he suggests using the weights of the best working model (i.e. save the model each time the validation error is checked).
I don't need the saved model, I only need the results for my work. So for me the proposed early stopping by Goodfellow would mean I'd just take the highest validation accuracy I've reached for my final result? Somehow this doesn't seem legit. I don't have this information in a real-world situation when there is no development set. But in that case, what is the reason to use early stopping in the first place? Determining the number of epochs by e.g. averaging the number of epochs for the folds and use it for the test run later on?