# Maximum likelihood estimators and overfitting

In his book, Bishop claims that overfitting is caused by an unfortunate property of the Maximum likelihood estimator. I dont really understand how the MLE relates to overfitting.

To me, roughly, overfitting is related to the model complexity, i.e. the more parameter I have, the more my model tends to overfit (i.e., to model the random noise).

Maximum likelihood estimation, however, is just a way to estimate statistics from my sample (or training set). As far as I understand it, it does not regulate the number of parameters whatsover and therefore I do not see the connection between MLE and overfitting.

Also, Maximum likelihood estimators often are biased. But biased models rather tend to underfit than overfit.

1.) How are these two things related and how does the MLE induce overfitting?

2.) Is there a "mathematical" justification, i.e., is it possible to show in terms of formulae, how these two things are connected? (as a similar question was already asked here but only with rather handwaving answers)

3.) Which "unfortunate property" of MLE is it, that Bishop claims to be the reason for overfitting?

• 1. What book? 2. Can you provide the quote? 3. It seems extremely unlikely to me that Bishop would write such a thing without explaining it anywhere. Are you sure he's not referring to something explained elsewhere? – jbowman Nov 13 '18 at 18:46
• In the book "Pattern recognition and Machine learning." The quote is from p.147, which is section 3.2. He also refers to that problem in section 1.1, 1.2.4 and 1.3. But he never gets more precise than that, otherwise I would not have asked the question – guest1 Nov 13 '18 at 18:50

The key to understanding Bishop's statement lies in the first paragraph, second sentence, of section 3.2: "... the use of maximum likelihood, or equivalently least squares, can lead to severe over-fitting if complex models are trained using data sets of limited size".

The problem comes about because no matter how many parameters you add to the model, the MLE technique will use them to fit more and more of the data (up to the point at which you have a 100% accurate fit), and a lot of that "fit more and more of the data" is fitting randomness - i.e., overfitting. For example, if I have $$100$$ data points and am fitting a polynomial of degree $$99$$ to the data, MLE will give me a perfect in-sample fit, but that fit won't generalize at all well - I really cannot expect to achieve anywhere near a 100% accurate prediction with this model. Because MLE is not regularized in any way, there's no mechanism within the maximum likelihood framework to prevent this overfitting from occurring. This is the "unfortunate property" referred to by Bishop. You have to do that yourself, by hand, by structuring and restructuring your model, hopefully appropriately. Your statement "... it does not regulate the number of parameters whatsoever..." is actually the crux of the connection between MLE and overfitting!

Now this is all well and good, but if there were no other model estimation approaches that helped with overfitting, we wouldn't be able to say that this was an unfortunate property specifically of MLE - it would be an unfortunate property of all model estimation techniques, and therefore not really worth discussing in the context of comparing MLE to other techniques. However, there are other model estimation approaches - Lasso, Ridge regression, and Elastic Net, to name three from a classical statistics tradition, and Bayesian approaches as well - that do attempt to limit overfitting as part of the estimation procedure. One could also think of the entire field of robust statistics as being about deriving estimators and tests that are less prone to overfitting than the MLE. Naturally, these alternatives do not eliminate the need to take some care with the model specification etc. process, but they help - a lot - and therefore provide a valid contrast with MLE, which does not help at all.

• Are ridge Regression and Lasso Regression proper independent estimators themselves? I thought these were just extensions of least squares ... – guest1 Nov 13 '18 at 21:34
• What do you mean by "proper independent estimators"? The two estimators, as typically implemented, have their own optimization algorithms, they don't just modify the least squares solution for $\beta$. – jbowman Nov 13 '18 at 22:40
• I mean that they stand on their own ground, i.e. are not only a special case of least squares. I thought that least squares and maximum likelihood were the only estimators being used. Ridge regression never occured to me as an estimator but only as an extension of the loss function of linear regression to somewhat control overfitting. – guest1 Nov 14 '18 at 7:03
• I have then another question. In section 1.2.4, p.27, he says "In particular, we shall show that the maximum likelihood approach systematically underestimates the variance of the distribution. This is an example of a phenomenon called bias and is related to the problem of over-fitting encountered in the context of polynomial curve fitting." I.e., he suggests that the problem of maximum likelihood causing overfitting is due to the fact that it produces bias estimates. But to me, bias rather causes underfitting, no? – guest1 Nov 14 '18 at 7:06
• Bishop mentions the bias introduced by the maximum likelihood on another occasion again, as the source of the problem of MLE. So is it really the bias introduced by the MLE that somehow causes overfitting? "Historically various ‘information criteria’ have been proposed that attempt to correct for the bias of maximum likelihood by the addition of a penalty term to compensate for the over-fitting of more complex models." p.33 – guest1 Nov 14 '18 at 11:05

Bishop may have been talking about conditional probability tables, or "group by" aggregations of the data often used in Bayesian networks. The MLE for these probabilities overfits in that it is too particular to the training data and may not generalize. This becomes especially true once you start adding variables to that grouping and slicing your data very thin. The MLEs for these group probabilities need some kind of regularization by means of a prior distribution, pooling, or some other method.

MLE is a technique used to estimate parameters from a model. Not only the parameter estimates of a distribution you assume on a specific variable, but just as well the parameters of any model that contains distributional assumptions such as Generalized Linear Models or Tree-based models with a conditional distribution assumption on the dependent. While a more complex model can increase the training likelihood, it does not necessarily mean the model will have an equally high likelihood on new unseen examples. Therefore, we should look at the likelihood on a separate test set, or penalize the model selection criteria for model complexity (such as AIC, BIC).

maximum likelihood can exhibit severe over-fitting for data sets that are linearly separable. This arises because the maximum likelihood solution occurs when the hyperplane corresponding to σ = 0.5, equivalent to w^Tφ = 0, separates the two classes and the magnitude of w goes to infinity and then causes highly fluctuation in decision boundary. In this case, the logistic sigmoid function becomes infinitely steep in feature space, corresponding to a Heaviside step function, so that every training point from each class k is assigned a posterior probability p(C_k|x) = 1.