4
$\begingroup$

In Bayesian Model Selection, we first compute the evidence: $$ p(D|m) = \int_{\theta} p(D|\theta) p(\theta|m) d\theta $$

Then, we select the model that maximizes, which is a MAP estimator of the model. $$ p(m|D) = \frac{p(D|m) p(m)}{\sum_m p(m,D)} $$

If we assume the prior of the models $p(m)$ is uniform, this equates to selecting the model that maximizes $p(D|m)$. This is all fine.

But here is my question: how do we draw the line between parameters $\theta$ and model $m$. A more complex model can include simpler models. For example, a polynomial with 10 degrees of freedom clearly includes a polynomial with 3 degrees of freedom as we can just set the appropriate coefficients to 0 to get to the latter. Conversely, we can restrict parameters to a small set and treat each set as a model. At an extreme, we can say that every unique parameter $\theta$ value is a new model. In this sense, $$ p(D|m) = p(D|\theta) $$

But here is the problem. We know that complex models tend to have larger likelihoods. So if we think of each unique parameter is a model, then the model that maximizes $p(D|m) = p(D|\theta)$ clearly overfits, defeating the purpose of model selection.

Of course, this is an extreme case to illustrate my point. But I feel that the line between model and parameters is blurred. We can have a lot of models with restrictive parameters, or fewer models with parameters taking wider ranges. And how we draw the line seems to affect the result of Bayesian model selection. In particular, if we tilt the line towards more models with restrictive parameters, it seems more prone to overfitting.

So in practice, how do people draw the line between models and parameters and go about Bayesian model selection?

A side question:

In both of the sources I cited below, evidence is computed as: $$ p(D|m) = \int_{\theta} p(D|\theta) p(\theta|m) d\theta $$ However, I wonder if the likelihood with regard to $\theta$ should be $p(D|\theta, m)$ instead of $p(D|\theta)$. I am not sure why it is considered not dependent on $m$.

References:

  1. http://alumni.media.mit.edu/~tpminka/statlearn/demo/
  2. MURPHY, KEVIN P. MACHINE LEARNING: a Probabilistic Perspective. MIT Press, 2012.
$\endgroup$
1
  • $\begingroup$ (+1) This is a very engaging set of questions! $\endgroup$
    – Xi'an
    Jan 28, 2021 at 20:07

1 Answer 1

3
$\begingroup$

In Bayesian model choice, the family of models under comparison is $$\mathfrak F=\left\{p(\cdot|\theta_m,m)\,; m=1,\ldots,M,\,\theta_m\in\Theta_m \right\}$$ and the parameter of a member of this family is the pair $(m,\theta)$. I index $\theta_m$ by $m$ to stress that the parameter is model-dependent. Given a prior $\pi(\cdot)$ on the pair $(m,\theta)$, the posterior weight $p(m|D)$ is the marginal in $m$ of the joint posterior $\pi(m,\theta_m|D)$. The most likely model associated with $p(m|D)$ is a Bayesian decision procedure associated this posterior and a $0-1$ loss function, also known as MAP estimator, but only a Bayesian decision procedure rather than the outcome of a Bayesian analysis.

Concerning the worries about over-fitting, the evidence is responding to a larger parameter space by penalising a more poor fit thanks to the integration over this larger parameter space. This is connected with the intuition that distributions in larger dimensions are more concentrated, hence are more penalised when integrated out. The ratio of evidences known as Bayes factors are consistent tools for selecting a model and the first order approximation of the evidence known as BIC, Schwarz' criterion or the "Bayesian" information criterion, $\text{BIC}_m \approx −2\log p(D|m)$, is consistent as well.

enter image description here

This property of the (Bayesian) evidence to automatically penalise the more complex models is often called Ockham's razor, even though I personally dislike the term as non-quantitative. Some Bayesian papers promote a more balanced perspective, in comparison.

$\endgroup$
3
  • $\begingroup$ Thanks for the response. Here is my concern. Let's look at $m=1$. Let $\Theta_1$ be the set of all valid values of $\theta_1$. Now let's break $\Theta_1$ into two disjoint subsets $\Theta_1'$ and $\Theta_2'$.. We then call model 1 with $\theta_1$ restricted on $\Theta_1'$ the new model 1, and model 1 with $\theta_1$ restricted on $\Theta_2'$ the new model $m+1$. So now we have $m+1$ models instead of $m$ models, and going through the same same model selection process with these $m+1$ models may give us different results than with $m$ models. $\endgroup$ Jan 28, 2021 at 20:34
  • $\begingroup$ The issue is that there seems no clear line between a model and a parameter. We can divide the range of $\theta_1$ into $n$ sets and make $n$ new models.At an extreme, each subset can contain only one point, and so each parameter is a new model. $\endgroup$ Jan 28, 2021 at 20:36
  • 2
    $\begingroup$ First, I agree there is no strict boundary between a parameter and a model index. The separation and the outcome depend very much on the primary definition of the data model. If fixing $\theta$ to a specific value makes a model of interest, then it should be singled out as a new model. The outcome and purpose of the analysis should be stated from the start. $\endgroup$
    – Xi'an
    Jan 28, 2021 at 20:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.