8
$\begingroup$

I've learned that over-fitting can be detected by plotting the training error and the testing error versus the epochs. Like in:

enter image description here

I've been reading this blogpost where they say the neural network, net5 is over-fitting and they provide this figure:

enter image description here

Which is strange to me, since the validation and training error of net5 keeps dropping (but slowly).

Why would they claim it is over fitting ? Is it because the validation error is stagnating ?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
11
$\begingroup$

Overfitting is not only when test error increases with iterations. We say that there is overfitting when the performance on test set is much lower than the performance on train set (because the model fits too much to seen data, and do not generalize well).

In your second plot we can see that performances on test sets are almost 10 times lower than performances on train sets, which can be considered as overfitting.

It's almost always the case that a model performs better on the training set than on test set, since the model has already seen the data. However, a good model should be able to generalize well on unseen data, and then to reduce the gap between performances on train and test sets.

Your first example of overfitting can be solved by early stopping for example. Your second example can be solved by regularization, by corrupting input, etc.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Why Overfitting is bad in that case? We can see it perform better on the test set so generalize better right? $\endgroup$
    – Fractale
    Commented Nov 14, 2018 at 10:06
  • 1
    $\begingroup$ @Fractale There are many other aspects to consider beyond the training step. For example, another set of hyperparameters may result in better test error and worse training error (stronger regularization). Therefore, such a configuration would result in less overfitting. "Over"-fitting always implies a comparison. Changing something such that it results in considerably better training error but worse or not significantly better test error is overfitting of the training examples, compared to the original setting. The "change" can be anything: the number of training iterations, hyperparams etc. $\endgroup$
    – isarandi
    Commented Dec 9, 2018 at 19:23

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.