Is reporting the test error of the model with the smallest validation error a sound way of doing Early Stopping Regularization?

Question

I wanted to use Early Stopping regularization to get the best generalization of my model trained via an iterative algorithm (e.g. some variant of SGD on a NN). I was thinking of implementing this idea as following:

1. Run SGD for N iterations
2. Record train, validation (cv) and test error for each epoch (even possible save the current model each epoch or even just the one with best cv)
3. Then report the test error (and return model) of the epoch with the lowest cross-validation.

Is this a sound way (maybe even shortcut) to implementing Early Stopping Regularization?

I've heard of other methods like track K cross-validation errors, etc but under the condition of sufficient memory space and patience for large N, I thought that maybe this could be a sound method to implement the idea instead.

---

The question is. If I train a NN for 1000 iterations and want to do early stopping. Can I just look at all the validation errors from all steps, note which one is the smallest validation error and then report the models test error relating to that step. Obviously training is only done with the train. The train data is for training, validation for choosing when to early stop and the test for reporting a test error that is not underestimated.

---

I don't understand why the order based on time/iteration even matters honestly, it seem arbitrary.

Answer 1

What you call "cross-validation error" should be the "validation error", i.e. the error computed on the validation set, which is a set of labeled samples that appear in neither the training set (otherwise the early stop might not be that early, and you'll overfit) nor the test set (forbidden).