In Bishop's PRML book, he says that, overfitting is a problem with Maximum Likelihood Estimation (MLE), and Bayesian can avoid it.

But I think, overfitting is a problem more about model selection, not about the method used to do parameter estimation. That is, suppose I have a data set $D$, which is generated via $$f(x)=sin(x),\;x\in[0,1]$$, now I might choose different models $H_i$ to fit the data and find out which one is the best. And the models under consideration are polynomial ones with different orders, $H_1$ is order 1, $H_2$ is order 2, $H_3$ is order 9.

Now I try to fit the data $D$ with each of the 3 models, each model has its paramters, denoted as $w_i$ for $H_i$.

Using ML, I will have a point estimate of the model parameters $w$, and $H_1$ is too simple and will always underfit the data, whereas $H_3$ is too complex and will overfit the data, only $H_2$ will fit the data well.

My questions are,

1) Model $H_3$ will overfit the data, but I don't think it's the problem of ML, but the problem of the model per se. Because, using ML for $H_1,H_2$ doesn't result into overfitting. Am I right?

2) Compared to Bayesian, ML does have some disadvantages, since it just gives the point estimate of the model parameters $w$, and it's overconfident. Whereas Bayesian doesn't rely on just the most probable value of the parameter, but all the possible values of the parameters given the observed data $D$, right?

3) Why can Bayesian avoid or decrease overfitting? As I understand it, we can use Bayesian for model comparison, that is, given data $D$, we could find out the marginal likelihood (or model evidence) for each model under consideration, and then pick the one with the highest marginal likelihood, right? If so, why is that?

  • $\begingroup$ @nbro, sorry it's been a while, I don't remember clearly, but maybe it meant Bayesian could somehow mitigate the overfitting issue, not avoid completely. $\endgroup$
    – avocado
    Commented Apr 16, 2020 at 10:47

3 Answers 3


Optimisation is the root of all evil in statistics. Any time you make choices about your model$^1$ by optimising some suitable criterion evaluated on a finite sample of data you run the risk of over-fitting the criterion, i.e. reducing the statistic beyond the point where improvements in generalisation performance are obtained and the reduction is instead gained by exploiting the peculiarities of the sample of data, e.g. noise). The reason the Bayesian method works better is that you don't optimise anything, but instead marginalise (integrate) over all possible choices. The problem then lies in the choice of prior beliefs regarding the model, so one problem has gone away, but another one appears in its place.

$^1$ This includes maximising the evidence (marginal likelihood) in a Bayesian setting. For an example of this, see the results for Gaussian Process classifiers in my paper, where optimising the marginal likelihood makes the model worse if you have too many hyper-parameters (note selection according to marginal likelihood will tend to favour models with lots of hyper-parameters as a result of this form of over-fitting).

G. C. Cawley and N. L. C. Talbot, Over-fitting in model selection and subsequent selection bias in performance evaluation, Journal of Machine Learning Research, 2010. Research, vol. 11, pp. 2079-2107, July 2010. (pdf)

  • 1
    $\begingroup$ +1, thank you so much, I'll read your paper and see if I have any further questions, ;-) $\endgroup$
    – avocado
    Commented Jan 22, 2014 at 1:10
  • 1
    $\begingroup$ Just to note here that optimization can usually be thought of as approximately integrating - the Laplace method is an example of this. Optimising usually fails when it isn't a good approximation to integrating - hence why REML is usually better than ML. $\endgroup$ Commented Jan 22, 2014 at 12:35
  • $\begingroup$ @probabilityislogic, I am not sure I understand, ML is a bit like MAP, there is no integration performed. Using the Laplace approximation (in the way I have seen it used) is optimising in the sense that you optimise an approximation to the function your want to integrate and integrate that instead, but there is still integration going on. $\endgroup$ Commented Jan 22, 2014 at 16:35
  • 1
    $\begingroup$ @dikran marsupial - Perhaps a better way to explain it is that integration is often well approximated by estimating a parameter by ML, and constraining that parameter to be equal its MLE. The Laplace approximation provides a "correction factor" to this intuition - in the same way that REML does. $\endgroup$ Commented Jan 22, 2014 at 22:17
  • $\begingroup$ @probabilityislogic thanks for the reply, I'll give it some thought! $\endgroup$ Commented Jan 23, 2014 at 9:16

As a general response, if you're using "least squares" type regression models there really isn't much difference between bayes and ML, unless you use an informative prior for the regression parameters. In response to specifics:

1)$ H_9 $ won't necessarily overfit the data - only when you have close to 9 observations. If you had 100 observations, most of the supposedly "overfitted" coefficients will be close to zero. Also $ H_1 $ would almost always result in "underfitting" - as there would be clear curvature missed

2) This is, not true for "linear" like polynomial expansions ("linear" meaning linear with respect to parameters, not $ x $). ML estimates for least squares are identical to posterior means under non informative priors or large sample sizes. In fact you can show that ML estimates can be thought of as "asymptotic" posterior means under a variety of models.

3) The Bayesian approach can avoid overfitting only for proper priors. This operates in a similar manner to penalty terms you see in some fitting algorithms. For example, L2 penalty = normal prior, L1 penalty = laplace prior.

  • $\begingroup$ upvoted, and you're right that with more observations at hand, $H_9$ won't overfit. But am I right with the claim that overfitting is a problem of choosing the wrong model, not the ML per se? And we can use Bayesian in model selection, but we can't do that with ML, can we? $\endgroup$
    – avocado
    Commented Jan 19, 2014 at 12:39
  • $\begingroup$ Surely all choices of H here will be the wrong model, other than $H_\infty$. The problem is the error in estimating the parameters of the model, which has both bias and variance components. If you choose the model using a Bayesian criterion, you can still over-fit that as well (I'll add a reference to support that in my answer). $\endgroup$ Commented Jan 21, 2014 at 17:56
  • $\begingroup$ @loganecolss - I think $ H_9 $ here would be closer to the truth than any others. Overfitting is more closely linked with the sample size and the type of model structure that it can provide support for (sometimes called the "sure thing" model). $\endgroup$ Commented Jan 22, 2014 at 12:44

Basically, what you're doing by increasing the degrees of your polynomials is increasing the number of parameters or degrees of freedom of your model space, ie. its dimension. The more parameters you add, the more the model can fit the training data easily. But this also depends heavily on the number of observations. Your models $H_1$ and $H_2$ might just as well overfit the training data if the number of observations is low, just as $H_3$ may not overfit at all if the number of training instances is large enough.

For example, let's grossly exaggerate and suppose you are given only $2$ training examples, than even $H_1$ will always overfit your data.

The advantage of imposing priors for instance through regularisation is that the parameters are either shrunk to zero or some other predefined value (you can even add parameters to "tie" the coefficients together if you like), and thus you are implicitly constraining the parameters and reducing the "freedom" of your model to overfit. For example, using the lasso (ie. $l^1$ regularisation or equivalently a Laplace prior) and tuning the corresponding parameter (using 10x cross validation for example) will automatically get rid of the surplus parameters. The Bayesian interpretation is similar : by imposing priors, you are constraining your parameters to some more probable value, inferred from the overall data.

  • $\begingroup$ A simple hypothesis (e.g. h1, h2) with insufficient training samples would be an example of under fitting (for cv) and not over fitting due to model bias on the few training examples given. $\endgroup$
    – yekta
    Commented Oct 9, 2016 at 9:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.