When to stop training of neural network when validation loss is still decreasing but gap with training loss is increasing?

Question

During training of CNNs, I often come across this case for training and validation loss :
X axis is epochs, Y axis is cross entropy loss.

I would like to keep the "best model", meaning the one which generalizes the best (I wish to use it for fine-tuning on another dataset).
So up until now, I wass keeping the model at the "green line" point : the one with the lowest validation loss. However, I'm concerned because before that point, the gap between training loss and validation loss has increased, which indicates overfitting. So should I rather be keeping the model at the "red line", before overfitting gets too much out of hand ?

Answer 1

As long as your validation loss is continuing to decrease, your model is continuing to perform better in a generalized setting.

A growing gap between training and validation performance does mean that your model is treating more and more of the noise in your training set as real signal. However, the fact that your validation loss is decreasing means that the model is taking even more advantage of true signal. In some sense, it's gaining more generalizability than it's losing.

I would agree with your first instincts, and choose the model at the iteration indicated with the green line.