I have made some research about overfitting and underfitting, and I have understood what they exactly are, but I cannot find the reasons.
What are the main reasons for overfitting and underfitting?
Why do we face these two problems in training a model?
I'll try to answer in the simplest way. Each of those problems has its own main origin:
Overfitting: Data is noisy, meaning that there are some deviations from reality (because of measurement errors, influentially random factors, unobserved variables and rubbish correlations) that makes it harder for us to see their true relationship with our explaining factors. Also, it is usually not complete (we don't have examples of everything).
As an example, let's say I am trying to classify boys and girls based on their height, just because that's the only information I have about them. We all know that even though boys are taller on average than girls, there is a huge overlap region, making it impossible to perfectly separate them just with that bit of information. Depending on the density of the data, a sufficiently complex model might be able to achieve a better success rate on this task than is theoretically possible on the training dataset because it could draw boundaries that allow some points to stand alone by themselves. So, if we only have a person who is 2.04 meters tall and she's a woman, then the model could draw a little circle around that area meaning that a random person who is 2.04 meters tall is most likely to be a woman.
The underlying reason for it all is trusting too much in training data (and in the example, the model says that as there is no man with 2.04 height, then it is only possible for women).
Underfitting is the opposite problem, in which the model fails to recognize the real complexities in our data (i.e. the non-random changes in our data). The model assumes that noise is greater than it really is and thus uses a too simplistic shape. So, if the dataset has much more girls than boys for whatever reason, then the model could just classify them all like girls.
In this case, the model didn't trust enough in data and it just assumed that deviations are all noise (and in the example, the model assumes that boys simply do not exist).
Bottom line is that we face these problems because:
Overfitting is when a model estimates the variable you are modeling really well on the original data, but it does not estimate well on new data set (hold out, cross validation, forecasting, etc.). You have too many variables or estimators in your model (dummy variables, etc.) and these cause your model to become too sensitive to the noise in your original data. As a result of overfitting on the noise in your original data, the model predicts poorly.
Underfitting is when a model does not estimate the variable well in either the original data or new data. Your model is missing some variables that are necessary to better estimate and predict the behavior of your dependent variable.
The balancing act between over and underfitting is challenging and sometimes without a clear finish line. In modeling econometrics time series, this issue is resolved pretty well with regularization models (LASSO, Ridge Regression, Elastic-Net) that are catered specifically to reducing overfitting by respectively reducing the number of variables in your model, reducing the sensitivity of the coefficients to your data, or a combination of both.
Perhaps during your research you came across the following equation:
Error = IrreducibleError + Bias² + Variance.
Why do we face these two problems in training a model ?
The learning problem itself is basically a trade-off between bias and variance.
What are the main reasons for overfitting and underfitting ?
Short: Noise.
Long: The irreducible error: Measurement errors/fluctuations in the data as well as the part of the target function that cannot be represented by the model. Remeasuring the target variable or changing the hypothesis space (i.e. selecting a different model) changes this component.
Edit (to link to the other answers): Model performance as complexity is varied:
where errorD is the error over the entire distribution D (in practice estimated with test sets).
Almost all statistical problems can be stated in the following form:
Given the data $(y, x)$ find $\hat{f}$ which produces $\hat{y}=\hat{f}(x)$.
Make this $\hat{f}$ as close as possible to "true" $f$, where $f$ is defined as
$$y = f(x) + \varepsilon$$
The temptation is always to make $\hat{f}$ produce $\hat{y}$ which are very close to the data $y$. But when new data point arrives, or we use data which was not used to construct $\hat{f}$ the prediction may be way off. This happens because we are trying to explain $\varepsilon$ instead of $f$. When we do this we stray from "true" $f$ and hence when new observation comes in we get a bad prediction. This when overfitting happens.
On the other hand when we find $\hat{f}$ the question is always maybe we can get a better $\tilde{f}$ which produces better fit and is close to "true" $f$? If we can then we underfitted in the first case.
If you look at the statistical problem this way, fitting the model is always a balance between underfitting and overfitting and any solution is always a compromise. We face this problem because our data is random and noisy.
What are the main reasons for overfitting and underfitting ?
For overfitting, the model is too complex to fit the training data well. For underfitting, the model is too simple.
Why do we face these two problems in training a model ?
It is hard to pick the "just right" model and parameters for the data.
Overfitting and underfitting are basically inadequate explanations of the data by an hypothesized model and can be seen as the model overexplaining or underexplaining the data. This is created by the relationship between the model used to explain the data and the model generating the data. In our attempt to explain we cannot access the underlying model so our judgement is guided by another factor: the uncertainties or error bars.
When, in an attempt to fit all the variance, we use a model that is too complex, we are overfitting. This is created by us having free reign in the model choice and according too much importance to the error bars (or trying to explain all the variability, which is the same). When restricting ourselves to a model that is too simple to describe the data, and not assigning enough importance to the error bars (or not explaining the variability), we are underfitting.
How can one avoid these two? Information backed models (not derived from the data but from prior knowledge of the problem) and meaningful uncertainties.
In a nutshell, overfiitting appears as a consequence of patterns that appear in your training dataset but are not present on the entire population (they appeared out of luck) If your use a simple model (think for linear regression for instance), risk of overfitting is low, as the number of possible patterns it can detect is small and therefore the chance of one of those randomly showing in the sample is not that big either. An example of this may occur if you try to study correlations 1,000,000 variables on a population taking a 100-individual sample. Some of the features may randomly present a huge sample correlation despite being completely independent from one another
Another reason for overfitting is biased sampling (the "sample fake patterns" are there because the sample is not really random) For example, if you want to study the average size of a certain kind of mushroom by going out there and finding them in nature, you are likely to overestimate it (bigger mushrooms are easier to find)
Underfitting is, on the other hand, a quite simpler phenomenon. It can mean two very basic things: A) We do not have enough data for the model to learn the population pattern or B) Our model is not powerful enough to reflect it.
You can find a case of A if you have a phenomenon like $y = a*x + \epsilon$ where $\epsilon$ is a random variable with mean 0 and standard deviation 1000, and the actual value of a (the parameter you want to estimate) ia 1. If you don't take enough data, you may not even be able to distinguish a from 0 thus claiming y and x are uncorrelated/independent from one another.
B could occur if your model is way to simple, for example, if $y = x^2 + \epsilon$ and you try linear regression, well.... Good luck!
Short answer:
The main reason for overfitting is using a complex model when you have a small training set.
The main reason for underfitting is using a model that is too simple and can not perform well on the training set.
- Models with high capacity can overfit by memorizing properties of the training set that do not serve them well on the test set.
-Deep Learning book, Goodfellow et al.
The goal of machine learning is to train a model on the training set in hopes that it will perform just as well on the testing data. But does getting good performance on the training set always translate to a good performance on the testing set? It will not, because your training data is limited. If you have limited data, your model might find some patterns that work for that limited training set, but those patterns do not generalize to other cases (i.e. test set). This can be solved by either:
A- Providing a larger training set to the model to reduce the chance of having arbitrary patterns in the training set.
B- Using a simpler model so that the model will not be able to find those arbitrary patterns in the training set. A more complex model will be able to find more complicated patterns, so you need more data to make sure your training set is large enough to not contain arbitrary patterns.
(e.g. Imagine you want to teach a model to detect ships from trucks and you have 10 images of each. If most of the ships in your images are in the water, your model might learn to classify any picture with a blue background as a ship instead of learning how a ship looks. Now, if you had 10,000 images of ships and trucks, your training set is more likely to contain ships and trucks in a variety of backgrounds and your model can no longer just rely on the blue background.)
Underfitting occurs when the model is not able to obtain a sufficiently low error value on the training set.
Models with low capacity may struggle to fit the training set.
-Deep Learning book, Goodfellow et al.
Underfitting occurs when your model is just not good enough to learn the training set, meaning your model is too simple. Whenever we start solving a problem, we want a model which is at least able to get a good performance on the training set, and then we start to think of reducing overfitting. Generally, the solution to underfitting is pretty straight forward: Use a more complex model.
Consider an example where we have a hypothesis/model equation,
y=q*X+c,
where X=feature list, y=label and q and c are the coefficients that we have to trained.
If we come up with the coefficient values such that its large enough and start suppressing the features value(i.e. X) in such a case we always get the constant value of y, irrespective of any X value. This is called highly biased or underfit model.
Consider another complex hypothesis example,
y=q*X+r*sqr(X)+s*cube(X)+c, where q,r,s and c are the coefficients.
After identifying the best coefficients value it's possible that for the training data, we can get the minimum loss. It's only because we made our model so complex and tightly coupled that it behave very well with the training data. Whereas with the unseen data wwe can get pretty opposite results. This is called highly varianced or overfit model.
Biased model needs more complexity in Model selection whereas highly varianced model needs drop in complexity in model selection. Regularization technique can help us in identifying the proper level of model complexity and through this technique we can overcome with both the issues.
