2
$\begingroup$

Recently I had a discussion with one of my colleagues regarding the concept of overfitting. I have a model that shows the following training behavior. The first plot shows how the value of loss changes as the model is trained over several data folds.

enter image description here

The second plot shows the evolution of the F1 metric.

enter image description here

I think that these curves imply that the model is overfitting. We have clear divergence of the curves between the training and validation phases and a considerable gap between the performance of a model during these two phases. Also, the validation loss starts increasing after some point, while the training loss keeps improving.

However, the colleague argues that as far as the model still improves its quality during the training phase, it means that it hasn't yet overfitted the training dataset. So we should keep trying to overfit it by reaching the F1 score equal to 1.0 to make sure that the model complex enough and then start regularizing it.

I've never encountered this kind of approach before. My thinking is that as soon as you have a model that shows too good performance on the training dataset but fails on the validation, it means we hit the overfitting condition and should start thinking about possible regularization methods, or do some additional feature engineering.

Is my understanding of the concept of overfitting not correct? Are there different ways to approach this concept?

$\endgroup$
1
  • 1
    $\begingroup$ I'm going to guess "loss" is bad and "f1" is good and higher values of epoch represent accrual of data. Further I'm going to guess "train" are validation statistics in the same dataset as created the model and "valid" is some independent dataset. Nothing unusual about these graphics. The gap could be model overfitting or it could be volatility of early data. $\endgroup$
    – AdamO
    Feb 13 '20 at 21:48
2
$\begingroup$

The model should be complex enough to classify the data based on it's true (unknown) probability distribution. It doesn't need to be complex enough to memorise the whole training set, which is essentially what it's doing when you're trying to get a perfect training score without caring about the generalization error.

Your plots clearly show overfitting, regularizing could help and should be applied. If the training error stagnates and no overfit is observed, the complexity can also be increased.

$\endgroup$
1
  • $\begingroup$ Exactly, that's how I've always thought about it. Thank you! $\endgroup$
    – devforfu
    Feb 14 '20 at 9:47

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.