0
$\begingroup$

On p. 159 of Murphy's book Machine Learning: A Probabilistic Perspective, he is discussing model selection:

A more efficient approach [to model selection] is to compute the posterior over models

$$p(m \mid \mathcal{D}) = \frac{p(\mathcal{D} \mid m) p(m)}{\sum_{m \in \mathcal{M}}p(m, \mathcal{D})}$$

From this, we can easily compute the MAP model, $\hat{m} = \operatorname{argmax}p(m \mid \mathcal{D})$. This is called Bayesian model selection.

If we use a uniform prior over models, $p(m) \propto 1$, this amounts to picking the model that maximizes

$$p(\mathcal{D} \mid m) = \int p(\mathcal{D} \mid \boldsymbol{\theta})p(\boldsymbol{\theta} \mid m) d \boldsymbol{\theta}$$

I'm a bit confused about $p(\boldsymbol{\theta}\mid m)$ on the last line. If this doesn't depend on $\mathcal{D}$, then how are we getting $\boldsymbol{\theta}$? Are they saying that we have different priors on $\boldsymbol{\theta}$ for various models?

$\endgroup$
2
$\begingroup$

In the last line $p(\theta|m)$ is the prior with model $m$.

For instance, in a linear regression case, we might have two models, one with $n$ predictors and one with $n+1$ predictors. So the parameter vector $\theta$ would actually have a different length for each of those models. So there is a missing index $\theta^{(m)}$, which may be creating some confusion.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.