1
$\begingroup$

I have read many blogpost and articles about overfitting and underfitting, and I have, to some extent, understood what they exactly are, and different ways to overcome these two problems.

However, I am wondering which one of these two problems, under-fitting and overfitting, is the most common error in classification tasks? Or it depends on lots of other factors?

Thank you!

$\endgroup$
5
  • 3
    $\begingroup$ Someone would have to do a survey, & presumably have access to the ground truth in each case. $\endgroup$ – gung - Reinstate Monica Nov 25 '20 at 16:38
  • $\begingroup$ Thank you for your reply. But I don't understand, could you please elaborate on it. Thank you. $\endgroup$ – Aziz Qadeer Nov 25 '20 at 16:40
  • 3
    $\begingroup$ How could anyone possibly know which is more common without looking at all the models that have been built (or a representative sample of them) & seeing the proportions that were overfit, underfit, & about right? $\endgroup$ – gung - Reinstate Monica Nov 25 '20 at 16:47
  • $\begingroup$ aha, thank you. I heard that overfitting is probably the most common error that happens a lot. $\endgroup$ – Aziz Qadeer Nov 25 '20 at 16:59
  • 1
    $\begingroup$ It certainly could be. I'm not sure how to say with confidence. $\endgroup$ – gung - Reinstate Monica Nov 25 '20 at 17:03
1
$\begingroup$

gung is saying that in order to know which of the two is most common, we need to a) have access to lots of classification problems in the wild, and then b) have access to the true processes so that we can compare the model to the truth.

Think of it this way. When your niece asks if you like her picture of a frog, you know if the drawing is good or bad because you can compare it to a real frog. Without knowledge of what a frog looks like, you can't tell if the drawing is good or bad.

Same with models. Models will always either over or underfit because all models are approximations. To know which is more prevalent, we would need to know how the data were actually generated, which obviates the need for a model in the first place.

My intuition says that we almost always underfit (except in image problems, where I'm willing to bet we overfit). If a non-trivial proportion of classification tasks are tackled with logistic regression, there is no reason to believe the truth is linear on the log odds scale save mathematical convenience. Hence, underfitting. But that is just anecdote.

$\endgroup$
4
  • $\begingroup$ aha ok understood, Is there such a survey exists? Not for all classification algorithms, maybe a least for some of the popular algorithms e.g. SVM or CRF. $\endgroup$ – Aziz Qadeer Nov 25 '20 at 16:55
  • $\begingroup$ Is the premise of my question fallacious? $\endgroup$ – Aziz Qadeer Nov 25 '20 at 16:58
  • $\begingroup$ Note that models can be both overfit & underfit (Can overfitting and underfitting occur simultaneously?). I suspect both are quite common. I'm not sure how to answer which would be more often. In part, I think it would depend on exactly how "underfitting" is defined (see the comment I left to Stephan's answer there). $\endgroup$ – gung - Reinstate Monica Nov 25 '20 at 17:03
  • 1
    $\begingroup$ @QadeerAziz, I don't think the premise of your question is fallacious. I just don't know how to ground an answer. $\endgroup$ – gung - Reinstate Monica Nov 25 '20 at 17:04
0
$\begingroup$

I don't have hard evidence to back this up, but I would suspect that overfitting is the more prevalent problem. In an overfit classifier, you'll tend to get very good performance on your training data, but see that it does not generalize well to unseen data. An underfit classifier, on the other hand, won't perform very well on the training data, but won't perform much worse on unseen data, either.

In circumstances where the analyst is sloppy in their validation of results, overfit models are very exciting - they appear to be great models, but if you don't properly validate the results, you'll never notice that the model is actually junk. On the other hand, underfit models are not that exciting to begin with - they don't appear to be great models in any circumstance, so they appear less useful at first glance. One can imagine how this can lead to publication bias - overfit models can produce seemingly exciting output, get published, and then never have their performance reproduced on any other data (which is part of the replication crisis in science in general). Underfit models, on the other hand, don't perform that well to begin with, and are ostensibly less likely to be published.

The replication crisis in science strongly suggests to me that most published models are overconfident in their output, and simply do not generalize to independent datasets - this is basically the classic symptom of overfitting. Overfit models will have better performance on some data than underfit models, which plays directly into the well-known publication bias that positive results tend to be published more often than negative ones. Underfit models tend to produce poor but reproducible results, while overfit ones tend to produce good but irreproducible results - the lack of reproducibility in science and the bias toward publishing good results leads me to believe that overfit models get published more frequently than underfit ones.

$\endgroup$
1
  • $\begingroup$ aha, thank you for the long answer. $\endgroup$ – Aziz Qadeer Nov 26 '20 at 9:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.