The concept of overfitting is widely talked about but opinions on the determining factor for whether a network is starting to overfit seems to differ widely. I guess this is broadly combined with: "when should we stop training a network?"

The two common opinions I have seen are the following:

  1. A network is beginning to overfit when the validation loss no longer decreases
  2. A network is beginning to overfit when the training loss is less than the validation loss

To me, opinion 1 seems to be the correct intuition, but there are many with opinion 2.

I can see a case for 2 in a very hand-wavy way: Suppose your training loss is decreasing dramatically, while your validation loss is decreasing relatively slower. The model is quite clearly beginning to memorize the training data, and while this isn't negatively impacting generalisability (at least based on your validation set), your confidences are climbing and you really need to be sure your validation set is representative.

Is there a decent/definitive source on this that discusses this in an applied/real-world setting?

Edit: the answer linked in the reply to this question is not sufficient, it doesn’t discuss the issue in detail, provide any sort of reference nor contrast the two opinions that I have encountered.

2nd Edit: Mathematics for Machine Learning by Deisenroth, Faisal and Ong mention in section 8.2.3 that:

...if the test risk is much larger than the training risk, this is an indication of overfitting.

In that paragraph training risk is average loss on the training set, and test risk is that on the test set. This suggests this reference is in the second camp.

  • 1
    $\begingroup$ How to Identify Overfitting in Convolutional Neural network? $\endgroup$ Commented Dec 16, 2022 at 9:58
  • $\begingroup$ That answer does not provide the depth I am looking for and doesn’t discuss and contrast the two opinions that people have. $\endgroup$ Commented Dec 20, 2022 at 20:09
  • $\begingroup$ Regarding (2): validation error is usually higher than training error. $\endgroup$
    – Tim
    Commented Dec 21, 2022 at 18:27
  • $\begingroup$ Typically, yes, but that is affected by the level of dropout used (and I admit that’s ultimately an arbitrary decision) $\endgroup$ Commented Dec 21, 2022 at 21:25


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.