Neural network training without early stopping

Question

I was researching k-fold cross-validation, and read that one should train on k-1 of the k partitions and test on the remaining partition, and then repeat for each partition, averaging the results to get an estimate of model performance. This I understand; however, if there is no validation dataset, when should I stop the training (since early stopping is not possible, and so the model cannot simply be trained until generalisation performance starts to worsen)? I.e. should training be stopped after a set number of epochs, or when the gradient falls below a certain limit? Are there any tips for what these stopping parameters should be?

Answer 1

If there is no validation set, make one: from the training fold keep a few samples out and use them for early stopping.

Other options are:

1. Train until training error converges. If you have enough data and the model is regularized, you can avoid overfitting and this becomes a reliable measure.
2. Look for "Optimized Approximation Algorithm" paper; they describe a method for monitoring test performance by analyzing signal-to-noise ratio of the training error. I don't have a practical experience with the method though, so unfortunately I can't tell you how efficient it is.

Answer 2

It’s actually very simple. Just use $k-2$ folds for training, 1 for validation and 1 for testing. You probably never encountered this issue before, because fitting most of the “classical” statistical learning methods(SVMs, OLS, PLS, splines, GAMs, Gaussian Processes, etc.) corresponds to solving a convex optimization method: there is one and only one solution, and approximating it more or less accurately is “just” an issue of numerical analysis. Also, these methods don't have an overwhelming capacity, such as Deep Neural Networks do, and you don't use early stopping (as one of tools) to control overfitting. This is why you never had to use a training/validation/test split when doing cross-validation before.

Another possibility would be to skip the validation fold altogether. This requires that you use modern regulation methods in its place, such as Path-SGD, batch normalization and dropout, together with maybe older tools such as weight decay. These are usually applied together with early stopping, not in its place, but you May experiment and see what you get. It depends a lot on your architecture, also - I implicitly assumed that you are using a CNN, but you haven't told us anything about it.