4
$\begingroup$

During training of CNNs, I often come across this case for training and validation loss :Losses
X axis is epochs, Y axis is cross entropy loss.

I would like to keep the "best model", meaning the one which generalizes the best (I wish to use it for fine-tuning on another dataset).
So up until now, I wass keeping the model at the "green line" point : the one with the lowest validation loss. However, I'm concerned because before that point, the gap between training loss and validation loss has increased, which indicates overfitting. So should I rather be keeping the model at the "red line", before overfitting gets too much out of hand ?

$\endgroup$
  • 1
    $\begingroup$ A gap between training and testing loss does not indicate overfitting. This is a very common fallacy. $\endgroup$ – Matthew Drury Jul 13 '18 at 14:34
  • $\begingroup$ Could you please elaborate @MatthewDrury ? I know a small gap is nothing to worry about, but what does a widening gap like so indicate ? $\endgroup$ – Soltius Jul 13 '18 at 14:37
  • $\begingroup$ Sure, will do after i get off the bus! $\endgroup$ – Matthew Drury Jul 13 '18 at 15:01
1
$\begingroup$

As long as your validation loss is continuing to decrease, your model is continuing to perform better in a generalized setting.

A growing gap between training and validation performance does mean that your model is treating more and more of the noise in your training set as real signal. However, the fact that your validation loss is decreasing means that the model is taking even more advantage of true signal. In some sense, it's gaining more generalizability than it's losing.

I would agree with your first instincts, and choose the model at the iteration indicated with the green line.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.