Real calculation of Bayesian model selection

Question

I've been reviewing Bayesian model selection, where probability of data given the model is defined with the following equation: $$p(y|M_k) = \int p(y|\theta)p(\theta|M_k)d\theta$$ As I understand, $p(y|\theta)$ is simply the distribution of y under the parameters we've already defined. I'm a bit of confused by the term $p(\theta|M_k)$. I know that we can rewrite it with the Bayes rule: $$p(\theta|M_k) = p(\theta)p(M_k|\theta)$$ I suppose that we can find the value of $p(\theta)$ using the Beta probability distribution. But what is happening with $p(M_k|\theta)$? How do we estimate the probability of the model given the parameters? It feels like we should estimate the probability of the probability distribution given its parameters, so maybe it is a Dirichlet distribution problem?

I don't know what to do with this $p(M_k|\theta)$ term, will be very grateful if you can explain it to me.

Answer 1

It seems to me that each of the models you are comparing also come with a prior that is specific to that model. In a simple regression setting, for example, you are trying to choose between explaining $y$ with:

Model 1 ($M_1$), which has a single regressor $x_1$ and priors $f_1(\beta)$ and $g_1(\sigma)$. These priors are your $p(\theta|M_1$), with $\theta=(\beta,\sigma)$.

Model 2 ($M_2$), which has a single regressor $x_2$ and priors $f_2(\beta)$ and $g_2(\sigma)$. These priors are your $p(\theta|M_2$), with $\theta=(\beta,\sigma)$.

You can find a nice example in slides 42-46 here: https://drive.google.com/file/d/0B081GdveJIEkVmwzTUJzdk15WE0/view?usp=sharing

Answer 2

On the issue of model priors:

It is usually impossible to assign a specific prior to each of the $2^p=K$ models, where $p$ is the number of predictors. A conventional way to go about this problem is to elicit a non-informative prior to the models as it is hard to know anything specific about the model space. However, if you are certain about the probability inclusion of some variables then it is possible to convey this information in the set up.

The idea is to decentralise prior mass such that the MCMC can discover the majority of all good models. An example of a conventional set up for the model prior is as follows:

$\textrm{for}\quad M_k\in\mathcal{M} \quad \textrm{let} \quad M_k\sim{}Bin(p,\phi)\quad \textrm{where} \quad \phi\sim{}Beta(a,b) $

By letting $a=1$ you get $b=\frac{p-\mathbb{E}(M_k)}{\mathbb{E}(M_k)}$, thus one has only to specify the expected prior models size, although one wants this expectation to carry as little influence as possible.

You can use a tessellation model prior and a ridge prior for the coefficients if your design matrix is singular, that way you'll avoid centring prior mass around bad models, making it easier for the MCMC to discover good models.