I came across this in lecture but failed to understand why:
Why does maximum likelihood estimation have issues with over fitting? Given data X and you want to estimate parameter theta.
An example would be helpful.
$\begingroup$
$\endgroup$
3
-
2$\begingroup$ stats.stackexchange.com/questions/82664/… $\endgroup$– Nikolas RiebleFeb 10 '17 at 8:09
-
1$\begingroup$ While it is true that this question is somewhat underspecified &/or overly broad, the existence of a good & upvoted answer establishes that it is not too unclear to be answerable. I'm voting to leave open. $\endgroup$– gung - Reinstate MonicaFeb 11 '17 at 2:21
-
$\begingroup$ A common failure case of MLE is when the model is "too flexible" relative to the amount of data given, e.g., fitting a 3-component Gaussian mixture to two data points, or fitting a Bernoulli to a single coin toss. Collecting more data may fix this issue, but won't help when there is severe model misspecification (so MLE isn't even defined), e.g., when the data lies on a low-dimensional subset of $\mathbb{R}^d$ yet we insist on fitting a density model that has support over all of $\mathbb{R}^d$. Here are more details if you're interested. $\endgroup$– Yibo YangJan 19 at 7:24
Add a comment
|
$\begingroup$
$\endgroup$
3
Maximum likelihood does not tell us much, besides that our estimate is the best one we can give based on the data. It does not tell us anything about the quality of the estimate, nor about how well we can actually predict anything from the estimates.
Overfitting means, we are estimating some parameters, which only help us very little for actual prediction. There is nothing in maximum likelihood that helps us estimate how well we predict. Actually, it is possible to increase the likelihood beyond any bound, without increasing predictive accuracy at all.
To illustrate the last point about increasing likelihood without increasing predictive quality, let me give an example. Let's assume we want to predict the number of car crashes in the USA on a given day. As a predictor we only have the number of rocks analyzed by the curiosity rover on mars. Now, it seems highly unlikely that the predictor has any relation to the number of car crashes, but we can still generate a maximum likelihood model using that predictor. Maximum likelihood only tells us, it is the best we can do given the current dataset, even though this "best we can do" may still be total garbage. Since there is no relationship between the predictor and the number to be predicted, we cannot do anything except to overfit.
Now let's take this a bit further, and assume we want to further increase our maximum likelihood. We add the average distance of the planet Jupiter on that day as another predictor. Again this carries no predictive value. But our maximum likelihood for the model will increase. It cannot decrease, since we are still including the original predictor, so the model that just ignores the distance of Jupiter is a possible model, and this has the exact same likelihood as the previous model. So we are increasing likelihood without adding predictive value, i.e., we are overfitting.
Let's further assume somebody provides a model that is estimating the number of car crashes based on some reasonable predictors (number of cars driven on that day, whether the day is a holiday / weekend / weekday, etc.) and that model gives us a likelihood of $L$. Now we can carry our "astrological" model by just adding arbitrary figures derived from constellations of stars and planet. If we just add enough constellations, we can get our "astrological" model to have a maximum likelihood $L' > L$. Does that mean we should discard the well reasoned model and use the astrological one instead? Of course not.
This should show that overfitting is always present, unless we introduce some method to guard against overfitting.
-
2$\begingroup$ I really like your answer, but I miss how overfitting influences the external validity of prediction models. What I mean to say is, is that overfitting (due to finding maximum likelihood, as you explained), provides parameter estimates which are based on your data alone. As you never know whether you might have an 'unlucky' sample of associations between (true or false) predictors and outcome, you probably overestimated/overfit the coefficients in your model. This is why shrinkage methods (which combat extreme parameters) are often used to guard models against overfitting. $\endgroup$– IWSFeb 10 '17 at 8:39
-
2$\begingroup$ @IWS Yes, you are correct, I tried to include that in there at first, but then felt that it would complicate the simple explanation. As there are multiple causes for overfitting (e.g. your predictors may be meaningless/noisy, your sample may bad, or your model may tend to generalize badly) that all contribute to low predictive value, I find it hard to include all possible causes in a single explanation. Therefore, I decided to give an explanation based only on one of these causes. If you have some idea on how to integrate the others without overly complicating the issue just go ahead and edit. $\endgroup$– LiKaoFeb 10 '17 at 9:13
-
$\begingroup$
$\endgroup$
Some models are just too flexible: In these cases, maximum likelihood estimators can effectively "memorize" the data---signal and noise. Such considerations motivate reducing the flexibility of some models, through, for instance, some kind of regularization.
$\begingroup$
$\endgroup$
1
In my opinion, the reason of overfitting caused by the use of maximum likelihood method is that the values of parameters are estimated in the light of current data. You should derive the values of parameters in the light of future data instead of current data. If you are interested in this idea, please refer to the paper below.
Takezawa, K. (2012): "A Revision of AIC for Normal Error Models," Open Journal of Statistics, Vol. 2 No. 3, 2012, pp. 309-312. doi: 10.4236/ojs.2012.23038. http://www.scirp.org/journal/PaperInformation.aspx?paperID=20651
-
$\begingroup$ Unfortunately you failed to mention that the paper you are linking to is your own... $\endgroup$– HagbardJan 22 '21 at 14:38