I wanted to use Early Stopping regularization to get the best generalization of my model trained via an iterative algorithm (e.g. some variant of SGD on a NN). I was thinking of implementing this idea as following:
- Run SGD for N iterations
- Record train, validation (cv) and test error for each epoch (even possible save the current model each epoch or even just the one with best cv)
- Then report the test error (and return model) of the epoch with the lowest cross-validation.
Is this a sound way (maybe even shortcut) to implementing Early Stopping Regularization?
I've heard of other methods like track K cross-validation errors, etc but under the condition of sufficient memory space and patience for large N, I thought that maybe this could be a sound method to implement the idea instead.
The question is. If I train a NN for 1000 iterations and want to do early stopping. Can I just look at all the validation errors from all steps, note which one is the smallest validation error and then report the models test error relating to that step. Obviously training is only done with the train. The train data is for training, validation for choosing when to early stop and the test for reporting a test error that is not underestimated.
I don't understand why the order based on time/iteration even matters honestly, it seem arbitrary.