Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I have read many blogpost and articles about overfitting and underfitting, and I have, to some extent, understood what they exactly are, and different ways to overcome these two problems.
However, I am wondering which one of these two problems, under-fitting and overfitting, is the most common error in classification tasks? Or it depends on lots of other factors?
Thank you!
gung is saying that in order to know which of the two is most common, we need to a) have access to lots of classification problems in the wild, and then b) have access to the true processes so that we can compare the model to the truth.
Think of it this way. When your niece asks if you like her picture of a frog, you know if the drawing is good or bad because you can compare it to a real frog. Without knowledge of what a frog looks like, you can't tell if the drawing is good or bad.
Same with models. Models will always either over or underfit because all models are approximations. To know which is more prevalent, we would need to know how the data were actually generated, which obviates the need for a model in the first place.
My intuition says that we almost always underfit (except in image problems, where I'm willing to bet we overfit). If a non-trivial proportion of classification tasks are tackled with logistic regression, there is no reason to believe the truth is linear on the log odds scale save mathematical convenience. Hence, underfitting. But that is just anecdote.
I don't have hard evidence to back this up, but I would suspect that overfitting is the more prevalent problem. In an overfit classifier, you'll tend to get very good performance on your training data, but see that it does not generalize well to unseen data. An underfit classifier, on the other hand, won't perform very well on the training data, but won't perform much worse on unseen data, either.
In circumstances where the analyst is sloppy in their validation of results, overfit models are very exciting - they appear to be great models, but if you don't properly validate the results, you'll never notice that the model is actually junk. On the other hand, underfit models are not that exciting to begin with - they don't appear to be great models in any circumstance, so they appear less useful at first glance. One can imagine how this can lead to publication bias - overfit models can produce seemingly exciting output, get published, and then never have their performance reproduced on any other data (which is part of the replication crisis in science in general). Underfit models, on the other hand, don't perform that well to begin with, and are ostensibly less likely to be published.
The replication crisis in science strongly suggests to me that most published models are overconfident in their output, and simply do not generalize to independent datasets - this is basically the classic symptom of overfitting. Overfit models will have better performance on some data than underfit models, which plays directly into the well-known publication bias that positive results tend to be published more often than negative ones. Underfit models tend to produce poor but reproducible results, while overfit ones tend to produce good but irreproducible results - the lack of reproducibility in science and the bias toward publishing good results leads me to believe that overfit models get published more frequently than underfit ones.
