Training loss decreasing, Validation loss steady, where to stop?

Question

In the following training scenario (Orange: training loss, Blue: validation loss), what epoch is the best time for stopping the training? Validation is almost steady as we continue training, but training keeps decreasing. In other words, having two models where both have the same validation loss, is the one with the lower training loss better?

Answer 1

Here's what I'd do:

1. Try with different training/test splits. It might be idiosyncratic. Especially with so few epochs.
2. I assume that you're using different hyperparameters? Perhaps save the parameters and resume with a different set of hyperparameters. This comment really depends on how you're doing hyperparameter optimization.
3. Depending on how costly it is to train the model and evaluate it, consider bagging your models, akin to how a random forest operates. In others words, fit your model to many different train/test splits, and average the model outputs, either in terms of a majority classification vote, or an averaging of the predicted probabilities. In this case, I'd err on the side of a slightly overfit model, because of the way that averaging can mitigate overfitting. But I wouldn't train to death either, unless you're going to fit very very many neural nets, and somehow ensure that you're decorrelating them akin to the method of random subspaces from random forests.