Bayesian Inference Notation Confusion

Question

In Bayesian Inference the following notation is quite common:

$P(H|D) = \frac{P(D|H)P(H)}{P(D)}$

where $D$ is data and $H$ is hypothesis. Moreover $P(D)$ is represented as total probability.

$P(D) = \sum P(D|H)P(H)dH$

However in my opinion notation seems to be quite sloppy. It is confusing on two fronts:

1) In order for this to work H should be a partitioning of sample space of $D$ by definition of total probability. But if you look at this point of view, let's say $D\sim N(m,v)$ then its sample space is $-\infty$ to $\infty$. I don't see why a hypothesis should partition this sample space, and what is that partition exactly? Let's say that my hypothesis is $D \sim N(m',v)$. In reality there should be a selector function behind the scenes, but I cannot really put my finger on it.

2) I think that distribution of $D$ is independent of what I select as a hypothesis, because my selection doesn't have the power to change reality. Therefore $P(D|H) = P(D)$. And inference cannot be done.

My interpretation is that in reality the notation is trying to partition a sample space of hypotheses. But how that should be properly written in a rigorous notation I don't know. Or my interpretation is wrong.

Any help appreciated.

Answer 1

The easiest way to imagine this is to think of the sample space as the Cartesian product $\Omega = \mathcal{D} \times \mathcal{H}$, with measure $\mathbb{P}(D, H) := \mathbb{P}(H) \mathbb{P}(D|H)$. In this way, each $D$ or $H$ correspond to subsets of pairs in $\Omega$ - both $\mathcal{D}$ and $\mathcal{H}$ partition the sample space differently. In this simple scheme,

1. $\mathbb{P}(H)$ is the prior measure over the members of the partition $\mathcal{H}$,
2. $\mathbb{P}(D|H)$ is measure of $D$ relative to the subset $H$.

Note that a particular observation $D \in \mathcal{D}$ can be regarded as a subset $\{(D, H): H \in \mathcal{H} \}$ containing all the pairs consistent with $D$. This way of reasoning can easily be generalized to multiple observations. With this in mind, your answers are:

1) No, your hypotheses do not partition the set of observations.

2) Yes, the distribution of $D$ is independent of your model about the world represented by the sample space (, the $\sigma$-algebra) and the probability measure. Bayesian probability theory is just a model of how to systematically update your beliefs given your prior assumptions plus data.

Answer 2

Starting with your comment in part 2: $P(D|H)$ is not to be read as "the probability of getting this data given that I select hypothesis H". It is saying "probability of getting this given that hypothesis H is true". Clearly therefore $P(D|H) + P(D|¬H) = P(D)$

In other words, the partition can always in principle be between H is true and H is not true. By the law of the excluded middle, this must cover everything.

However, you are right to observe that if you are summing up different models that could explain the data, then there is no guarantee (or even reason to believe) that you have got "everything" covered (i.e. in practice it is usually impossible to know $P(D|¬H)$). That is why typically Bayesian analysis doesn't try to talk about the absolute probability of the model being correct given the data but only the relative probabilities of competing models.

So although I can in theory I can state:

$P(H_1 | D) = \frac{P(D|H_1) P(H_1)}{P(D)}$

$P(H_2 | D) = \frac{P(D|H_2) P(H_2)}{P(D)}$

When I do my analsis there is an implicit conditioning variable that is $M =$ "One of my models is correct". Perhaps being strict one should write:

$P(H_1 | D, M) = \frac{P(D|H_1) P(H_1|M)}{P(D|M)}$

$P(H_2 | D, M) = \frac{P(D|H_2) P(H_2|M)}{P(D|M)}$

Noting that $P(H) = P(H,M)$ since if H is true then M is true.

Now, however, notice that defined like this my priors must logically add up to 1. Often people don't have priors that add up to one. We could incorporate this by saying $P(H_1) = P(H_1|M)P(M)$. Writing our priors like this would then give us:

$P(H_1 | D, M) = \frac{P(D|H_1) P(H_1)}{P(D|M)P(M)}$

$P(H_2 | D, M) = \frac{P(D|H_2) P(H_2)}{P(D|M)P(M)}$

Of course none of this makes the slightest difference, because we are only going to compare the models, so we calculate:

$\frac{P(H_1 | D, M)}{P(H_2 | D, M)} = \frac{P(D|H_1,M) P(H_1)}{P(D|M)P(M)} \cdot \frac{P(D|M)P(M)}{P(D|H_2,M) P(H_2)} = \frac{P(D|H_1) P(H_1)} {P(D|H_2) P(H_2)}$

And all that intricacy cancels out and we are back at the original expressions.

Is it just laziness not writing the extra conditional variables that exist? Not really no. To me this is just another facet of the conditionality principle.

---

Edit

To take your example, suppose that $H_1$ is that $D \sim N(m,v)$ and $H_2$ is that $D \sim N(m', v)$ then clearly these two do not partition the space fully. Since we cannot evaluate $P(D|¬H_1 \cup H_2)$ we cannot work out the true $P(D)$ and so cannot ever get the absolute probability that either model is correct.

What we can do though is establish the ratio of those probabilities, since the unpartitioned part of the space cancels out (see above). This could then tell us that say, that $P(H_1|D)$ is 10 times more likely than $P(H_2|D)$.

In order to derive this, we had to write down expressions involving $P(D)$, even though we knew we'd never actually be able to calculate them.