How do we interpret model priors if models intersect?

Let $\Theta$ be some parameter space and $\Theta_1,\Theta_2,\dots,\Theta_s \subset \Theta$ be parameter subsets which represent competing models in some model selection procedure. There are no assumptions about containment or disjointness of the $\Theta_i$ made. The fundamental postulate of Bayesian model selection is that posterior probability of model $\Theta_i$ is computed according to Bayes rule $$P(\Theta_i | D) \propto P(\Theta_i) \int_{\Theta_i} L (\theta | D) dQ_i(\theta)$$ where $D$ stands for the data, $L$ the likelihood, $Q_i$ the conditional prior distribution of model $\Theta_i$, and $P(\Theta_i)$ the prior. My question is:

What do we mean by the prior probability $P(\Theta_i)$?

The prior should express our a priori belief of $\Theta_i$ being the right model in some sense, so one might come up with something like $\text{Prob}(\theta_0 \in \Theta_i)$, where $\theta_0$ is the true parameter generating the data (assuming one for now). However, this definition does not work because the true parameter may well be contained in several models.

Another way to ask the question: Assume I know the true parameter $\theta_0$, which prior should I choose?

Do we have to let go of the notion of an unknown true parameter? Is modeling with overlapping sets $\Theta_i$ inconsistent?

• Since you're using $\Theta_i$ to be equivalent to the $i$th mdoel, then $P(\Theta_i)$ is just the prior probability of model $i$: $P(M_i)$. Those model priors are up to you, but you must have $\sum_i P(M_i) = 1$. They represent your prior belief about the models before looking at the data. Is this not what you're asking about?
– user44764
Jun 15, 2014 at 20:54
• I'm not a practitioner of Bayesian statistics. So I guess my question really is: How to choose a prior? or: What does $P(M_i)$ mean? Another aspect: How would I transfer belief about the true parameter (say as a distribution on $\Theta$) into a (discrete) prior $P(\Theta_i)$? Jun 16, 2014 at 13:11
• I have added heavily to my answer.. hopefully I've found the issue we're getting stuck on.
– user44764
Jun 16, 2014 at 20:11

Let $Θ_i$ be some parameter space and $Θ_1,Θ_2,…,Θ_s⊂Θ$ be parameter subsets which represent competing models in some model selection procedure.

We generally do not equate different models as subsets of the parameter space. Observe that this undermines the premise of your question.. It would be misleading to equate parameter spaces with the models they correspond to, especially since the models may not even fully partition the parameter space.

Instead we consider different models $\mathcal{M}_1, \dots, \mathcal{M}_s$ and put a prior on these. That is, we specify $P(\mathcal{M}_1), \dots, P(\mathcal{M}_s)$, have them sum to one, and have all be nonnegative. Typically people are 'objective' and assign $P(\mathcal{M}_1) = \cdots = P(\mathcal{M}_s) = \frac{1}{s}$. This also simplifies some calculations.

Models are the different specifications of our prior belifs of the parameter(s) $\theta$. For example, we might have $\mathcal{M}_1$ be that $$\theta \mid \mathcal{M}_1 \sim \mathcal{N}(0,1),$$ and for $\mathcal{M}_2$ have $$\theta \mid \mathcal{M}_2 \sim \mathcal{N}(23472039487,209343204).$$ Perhaps in this light it will become clearer why we don't consider $P(\Theta_i)$ to be useful.

What do we mean by the prior probability $P(Θ_i)$?

If you call $P(\Theta_i) = P(\theta \in \Theta_i)$, then of what distribution is $\theta$ a random variable? If you say model $\mathcal{M}_i$, then $$P(\Theta_i) = P(\theta \in \Theta_i \mid \mathcal{M}_i) = 1.$$

When comparing models, what we really want to do is compare beliefs/explanations of what $\theta$ is through the mathematics of the models. Putting priors on the parameter spaces does not quantify our uncertainty about the value of $\theta$ in a way that is: (1) fully specified with our beliefs.. the shape and concentration of the models are very useful; or (2) mathematically possible.. overlapping and/or underspecified model spaces make these probabilities incoherent.

It is correct (and I hope you will find more agreeable) to look at the probabilities of various models rather than the parameter spaces these models cover. Consider the case where there are only two models $\mathcal{M}_1, \mathcal{M}_2$ with equal prior probability. Then consider the posterior model probability $$\begin{eqnarray*} P(\mathcal{M}_2 \mid \text{data}) &=& \frac{P(\mathcal{M}_2) P(\text{data} \mid \mathcal{M}_2)}{ P(\mathcal{M}_1) P(\text{data} \mid \mathcal{M}_1) + P(\mathcal{M}_2) P(\text{data} \mid \mathcal{M}_2)} \\ &=& \frac{ P(\text{data} \mid \mathcal{M}_2)}{ P(\text{data} \mid \mathcal{M}_1) + P(\text{data} \mid \mathcal{M}_2)} \\ &=& \frac{1}{1 + \text{BF}_{12}}, \end{eqnarray*}$$ where $\text{BF}_{12}$ is the Bayes factor (likelihood ratio) of the first model over the second. Observed that in this way $\theta$ is only used conditionally on the model through $$P(\text{data} \mid \mathcal{M}_i) = \int P(\text{data} \mid \theta) p(\theta \mid \mathcal{M}_i) d\theta.$$ In fact, it is certainly allowed to have models where we use different parameters in each case. For example, in $\mathcal{M}_1$ we might let the mean of a normal distribution vary in probability, but in $\mathcal{M}_2$ we might only let the variance vary in probability. In that sense, $P(\Theta_i)$ would be even more unfit to bear weight in the discussion. In this way this model comparison approach is extremely flexible.

Another way to ask the question: Assume I know the true parameter $θ_0$, which prior should I choose?

If you already know then why are you doing inference? Your prior in that case should be the constant a.k.a. $P(\theta = \theta_0) = 1$.

However, this definition does not work because the true parameter may well be contained in several models.

While this is typically the case in sampling-theory hypothesis testing, this is NOT the case in Bayesian model comparison. Again, this is because Bayesians do not compare models based entirely on regions of the parameter space. For example, consider a binomial likelihood for $y$ out of $N$ Bernoulli trials with two different models to consider: $\mathcal{M}_1 \iff P(\theta = \frac{3}{4}) = 1$ and $\mathcal{M}_2 \iff \theta \sim \text{Beta}(\frac{1}{2},\frac{1}{2})$, which happens to be the Jeffreys prior (though we are not doing any posterior updates here). Note that the parameter spaces for $\mathcal{M}_1$ and $\mathcal{M}_2$ have an intersection point of $\theta = \frac{3}{4}$. Suppose we put a uniform prior probability on the models (i.e., $P(\mathcal{M}_1) = P(\mathcal{M}_2) = \frac{1}{2}$).

We want to look at the Bayes factor for the two models given $y=30$ and $N=40$, which is $$\begin{eqnarray*} \text{BF}_{12} &=& \frac{P(y=30 \mid \mathcal{M}_1)}{P(y=30 \mid \mathcal{M}_2)} \\ &=& \frac{\text{Binom}(30 \mid 40, \frac{3}{4})}{\text{Beta-Binomial}(30 \mid 40, \frac{1}{2}, \frac{1}{2})} \\ &=& \frac{0.144}{0.0181} \\ &\approx& 8. \end{eqnarray*}$$ So we would conclude that the model $\mathcal{M}_1$ is $8$ times more likely than $\mathcal{M}_2$.. But nowhere did the model parameter spaces intersecting stop us from proceeding. Bayesian model comparison is a comparison of two models, not of two parameter spaces. The model distributions play a very important role. A surprisingly good discussion can be found in the wikipedia page for Bayes factors, where you can see how I copied the essential idea of one of their examples. The only textbook (out of 2, mind you) I have seen with much discussion of Bayes factors is Peter Hoff's First Course in Bayesian Statistical Methods, but most of his discussion is motivation. However, I haven't read Bayesian Data Analysis and it's a very strong and popular overview of Bayesian methods so it may have a more complete discussion. Hopefully someone who has read it can comment.

• Thanks for your answer. I had seen Bayes factors before, but with them we can only compare models pairwise, right? Still not sure what $P(\Theta_i)$ means... Jun 14, 2014 at 18:04
• @Thomas If you saw that in some context then perhaps you should describe the context.. As you've described it it just seems like you're referring to the prior specification for different models, and there's no problem with different models having overlapping (on the parameter space) priors as my answer indicates.
– user44764
Jun 14, 2014 at 18:09
• I have extended the question. I think the second bold face question may be what I'm really after. Jun 15, 2014 at 19:35
• OK, thanks for your efforts. I give up :) For the records, my question resulted from reading Section 5 in "Lectures on algebraic statistics" by Drton, Sullivant, Sturmfels. Jun 17, 2014 at 20:33
• @Thomas By their reckoning they have 'fixed the model'.. by which they mean that they have fixed the distribution and are only considering different choices of the parameters as different candidate models. I think what I have presented should be consistent with that viewpoint, though they are obviously looking at different model comparison tools. Perhaps you should re-ask the question/edit it to include the source of the discussion? I definitely don't want a check mark just because you 'give up', and that wouldn't help future questioners either
– user44764
Jun 17, 2014 at 20:47