Bayesian vs MLE, overfitting problem

Question

In Bishop's PRML book, he says that, overfitting is a problem with Maximum Likelihood Estimation (MLE), and Bayesian can avoid it.

But I think, overfitting is a problem more about model selection, not about the method used to do parameter estimation. That is, suppose I have a data set $D$, which is generated via $$f(x)=sin(x),\;x\in[0,1]$$, now I might choose different models $H_i$ to fit the data and find out which one is the best. And the models under consideration are polynomial ones with different orders, $H_1$ is order 1, $H_2$ is order 2, $H_3$ is order 9.

Now I try to fit the data $D$ with each of the 3 models, each model has its paramters, denoted as $w_i$ for $H_i$.

Using ML, I will have a point estimate of the model parameters $w$, and $H_1$ is too simple and will always underfit the data, whereas $H_3$ is too complex and will overfit the data, only $H_2$ will fit the data well.

My questions are,

1) Model $H_3$ will overfit the data, but I don't think it's the problem of ML, but the problem of the model per se. Because, using ML for $H_1,H_2$ doesn't result into overfitting. Am I right?

2) Compared to Bayesian, ML does have some disadvantages, since it just gives the point estimate of the model parameters $w$, and it's overconfident. Whereas Bayesian doesn't rely on just the most probable value of the parameter, but all the possible values of the parameters given the observed data $D$, right?

3) Why can Bayesian avoid or decrease overfitting? As I understand it, we can use Bayesian for model comparison, that is, given data $D$, we could find out the marginal likelihood (or model evidence) for each model under consideration, and then pick the one with the highest marginal likelihood, right? If so, why is that?

Answer 1

Optimisation is the root of all evil in statistics. Any time you make choices about your model$^1$ by optimising some suitable criterion evaluated on a finite sample of data you run the risk of over-fitting the criterion, i.e. reducing the statistic beyond the point where improvements in generalisation performance are obtained and the reduction is instead gained by exploiting the peculiarities of the sample of data, e.g. noise). The reason the Bayesian method works better is that you don't optimise anything, but instead marginalise (integrate) over all possible choices. The problem then lies in the choice of prior beliefs regarding the model, so one problem has gone away, but another one appears in its place.

---

$^1$ This includes maximising the evidence (marginal likelihood) in a Bayesian setting. For an example of this, see the results for Gaussian Process classifiers in my paper, where optimising the marginal likelihood makes the model worse if you have too many hyper-parameters (note selection according to marginal likelihood will tend to favour models with lots of hyper-parameters as a result of this form of over-fitting).

G. C. Cawley and N. L. C. Talbot, Over-fitting in model selection and subsequent selection bias in performance evaluation, Journal of Machine Learning Research, 2010. Research, vol. 11, pp. 2079-2107, July 2010. (pdf)

Answer 2

As a general response, if you're using "least squares" type regression models there really isn't much difference between bayes and ML, unless you use an informative prior for the regression parameters. In response to specifics:

1)$ H_9 $ won't necessarily overfit the data - only when you have close to 9 observations. If you had 100 observations, most of the supposedly "overfitted" coefficients will be close to zero. Also $ H_1 $ would almost always result in "underfitting" - as there would be clear curvature missed

2) This is, not true for "linear" like polynomial expansions ("linear" meaning linear with respect to parameters, not $ x $). ML estimates for least squares are identical to posterior means under non informative priors or large sample sizes. In fact you can show that ML estimates can be thought of as "asymptotic" posterior means under a variety of models.

3) The Bayesian approach can avoid overfitting only for proper priors. This operates in a similar manner to penalty terms you see in some fitting algorithms. For example, L2 penalty = normal prior, L1 penalty = laplace prior.

Answer 3

Basically, what you're doing by increasing the degrees of your polynomials is increasing the number of parameters or degrees of freedom of your model space, ie. its dimension. The more parameters you add, the more the model can fit the training data easily. But this also depends heavily on the number of observations. Your models $H_1$ and $H_2$ might just as well overfit the training data if the number of observations is low, just as $H_3$ may not overfit at all if the number of training instances is large enough.

For example, let's grossly exaggerate and suppose you are given only $2$ training examples, than even $H_1$ will always overfit your data.

The advantage of imposing priors for instance through regularisation is that the parameters are either shrunk to zero or some other predefined value (you can even add parameters to "tie" the coefficients together if you like), and thus you are implicitly constraining the parameters and reducing the "freedom" of your model to overfit. For example, using the lasso (ie. $l^1$ regularisation or equivalently a Laplace prior) and tuning the corresponding parameter (using 10x cross validation for example) will automatically get rid of the surplus parameters. The Bayesian interpretation is similar : by imposing priors, you are constraining your parameters to some more probable value, inferred from the overall data.