Why does maximum likelihood estimation have issues with over fitting?

Question

I came across this in lecture but failed to understand why:

Why does maximum likelihood estimation have issues with over fitting? Given data X and you want to estimate parameter theta.

An example would be helpful.

Answer 1

Maximum likelihood does not tell us much, besides that our estimate is the best one we can give based on the data. It does not tell us anything about the quality of the estimate, nor about how well we can actually predict anything from the estimates.

Overfitting means, we are estimating some parameters, which only help us very little for actual prediction. There is nothing in maximum likelihood that helps us estimate how well we predict. Actually, it is possible to increase the likelihood beyond any bound, without increasing predictive accuracy at all.

To illustrate the last point about increasing likelihood without increasing predictive quality, let me give an example. Let's assume we want to predict the number of car crashes in the USA on a given day. As a predictor we only have the number of rocks analyzed by the curiosity rover on mars. Now, it seems highly unlikely that the predictor has any relation to the number of car crashes, but we can still generate a maximum likelihood model using that predictor. Maximum likelihood only tells us, it is the best we can do given the current dataset, even though this "best we can do" may still be total garbage. Since there is no relationship between the predictor and the number to be predicted, we cannot do anything except to overfit.

Now let's take this a bit further, and assume we want to further increase our maximum likelihood. We add the average distance of the planet Jupiter on that day as another predictor. Again this carries no predictive value. But our maximum likelihood for the model will increase. It cannot decrease, since we are still including the original predictor, so the model that just ignores the distance of Jupiter is a possible model, and this has the exact same likelihood as the previous model. So we are increasing likelihood without adding predictive value, i.e., we are overfitting.

Let's further assume somebody provides a model that is estimating the number of car crashes based on some reasonable predictors (number of cars driven on that day, whether the day is a holiday / weekend / weekday, etc.) and that model gives us a likelihood of $L$. Now we can carry our "astrological" model by just adding arbitrary figures derived from constellations of stars and planet. If we just add enough constellations, we can get our "astrological" model to have a maximum likelihood $L' > L$. Does that mean we should discard the well reasoned model and use the astrological one instead? Of course not.

This should show that overfitting is always present, unless we introduce some method to guard against overfitting.

Answer 2

Some models are just too flexible: In these cases, maximum likelihood estimators can effectively "memorize" the data---signal and noise. Such considerations motivate reducing the flexibility of some models, through, for instance, some kind of regularization.

Answer 3

In my opinion, the reason of overfitting caused by the use of maximum likelihood method is that the values of parameters are estimated in the light of current data. You should derive the values of parameters in the light of future data instead of current data. If you are interested in this idea, please refer to the paper below.

Takezawa, K. (2012): "A Revision of AIC for Normal Error Models," Open Journal of Statistics, Vol. 2 No. 3, 2012, pp. 309-312. doi: 10.4236/ojs.2012.23038. http://www.scirp.org/journal/PaperInformation.aspx?paperID=20651