1
$\begingroup$

I wanted to use Early Stopping regularization to get the best generalization of my model trained via an iterative algorithm (e.g. some variant of SGD on a NN). I was thinking of implementing this idea as following:

  1. Run SGD for N iterations
  2. Record train, validation (cv) and test error for each epoch (even possible save the current model each epoch or even just the one with best cv)
  3. Then report the test error (and return model) of the epoch with the lowest cross-validation.

Is this a sound way (maybe even shortcut) to implementing Early Stopping Regularization?

I've heard of other methods like track K cross-validation errors, etc but under the condition of sufficient memory space and patience for large N, I thought that maybe this could be a sound method to implement the idea instead.


The question is. If I train a NN for 1000 iterations and want to do early stopping. Can I just look at all the validation errors from all steps, note which one is the smallest validation error and then report the models test error relating to that step. Obviously training is only done with the train. The train data is for training, validation for choosing when to early stop and the test for reporting a test error that is not underestimated.


I don't understand why the order based on time/iteration even matters honestly, it seem arbitrary.

$\endgroup$
1
$\begingroup$

What you call "cross-validation error" should be the "validation error", i.e. the error computed on the validation set, which is a set of labeled samples that appear in neither the training set (otherwise the early stop might not be that early, and you'll overfit) nor the test set (forbidden).

$\endgroup$
  • $\begingroup$ this seems more of a comment than a question. The question is. If I train a NN for 1000 iterations and want to do early stopping. Can I just look at all the validation errors from all steps, note which one is the smallest validation error and then report the models test error relating to that step. Obviously training is only done with the train. The train data is for training, validation for choosing when to early stop and the test for reporting a test error that is not underestimated. $\endgroup$ – Pinocchio Aug 8 '16 at 14:54
  • $\begingroup$ @Pinocchio what's the difference with the "traditional" early stopping? $\endgroup$ – Franck Dernoncourt Aug 8 '16 at 15:11
  • $\begingroup$ thats exactly my question :p ! Usually early stopping is described as actually stopping when the validation error starts to increase. This is easy to do by eye balling it, but in a computer one has to keep track of maybe the last k etc and then stop if it seem sensible. To avoid all that non-sense I was suggesting (in theory), just record everything during training and return the model and test error that resulted in the lowest validation error. $\endgroup$ – Pinocchio Aug 8 '16 at 15:39
  • $\begingroup$ @Pinocchio what's your early stopping criterion? $\endgroup$ – Franck Dernoncourt Aug 8 '16 at 16:10
  • 1
    $\begingroup$ @Pinocchio Yes, it is fine. That's the job of the validation set. (It would be cheating if you were reporting the smallest test set error.) $\endgroup$ – Franck Dernoncourt Aug 8 '16 at 22:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.