3
$\begingroup$

From my understanding of over- and underfitting, the two behaviors are not completely mutually exclusive. Overfitting can pretty much always be achieved if the model has enough capacitance (is able to capture the variance of the data) and is trained for long enough time. I have more issues with the concept of underfitting, meaning that the model has a bias problem and can't capture the complexity/variance of the data for the given task.

When evaluating if a model has an overtraining or undertraining problem, there is one case for which I don't know how to make an assessment: The case were a model starts overfitting very quickly (validation loss stops decreasing and starts increasing) but the training loss is still very high and performance is still very bad.

In this case the model is already capturing information that is not generalizable, but which is also not enough for getting a good fit to the training data. How is one supposed to combat this behavior? Is it still a sign of overfitting, for which we have to regularize and stop the model for learning non-generalizable information?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

A model that cannot really capture the structure of the data, i.e. a model that actually underfits, can still memorize something internally, that results in bad generalization. Although it is typically not considered as overfitting since your model is not able to capture the training data well, it actually emulates overfitting by spending its resources to memorize (maybe some portion of) your data. In such a case, you should stop the learning process, e.g. early-stopping for neural networks. But, more importantly, you should either update your model or your training algorithm (e.g. gradient descent method).

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ so although both underfitting and overfitting behaviors are present, we should categorize this situation as having a bias-problem (rather than a variance problem) and tackle it as if it were underfitting? Reduce regularization, increase model capacitance, add data, etc $\endgroup$
    – hirschme
    Mar 13, 2019 at 15:27
  • 2
    $\begingroup$ Yes, the important problem here is underfit. I believe, instead of reducing regularization, the problem is more likely to be solved by increasing the capacitance, and adding more data if available. $\endgroup$
    – gunes
    Mar 13, 2019 at 17:00

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.