# Bayesian Inference Notation Confusion

In Bayesian Inference the following notation is quite common:

$P(H|D) = \frac{P(D|H)P(H)}{P(D)}$

where $D$ is data and $H$ is hypothesis. Moreover $P(D)$ is represented as total probability.

$P(D) = \sum P(D|H)P(H)dH$

However in my opinion notation seems to be quite sloppy. It is confusing on two fronts:

1) In order for this to work H should be a partitioning of sample space of $D$ by definition of total probability. But if you look at this point of view, let's say $D\sim N(m,v)$ then its sample space is $-\infty$ to $\infty$. I don't see why a hypothesis should partition this sample space, and what is that partition exactly? Let's say that my hypothesis is $D \sim N(m',v)$. In reality there should be a selector function behind the scenes, but I cannot really put my finger on it.

2) I think that distribution of $D$ is independent of what I select as a hypothesis, because my selection doesn't have the power to change reality. Therefore $P(D|H) = P(D)$. And inference cannot be done.

My interpretation is that in reality the notation is trying to partition a sample space of hypotheses. But how that should be properly written in a rigorous notation I don't know. Or my interpretation is wrong.

Any help appreciated.

-

The easiest way to imagine this is to think of the sample space as the Cartesian product $\Omega = \mathcal{D} \times \mathcal{H}$, with measure $\mathbb{P}(D, H) := \mathbb{P}(H) \mathbb{P}(D|H)$. In this way, each $D$ or $H$ correspond to subsets of pairs in $\Omega$ - both $\mathcal{D}$ and $\mathcal{H}$ partition the sample space differently. In this simple scheme,

1. $\mathbb{P}(H)$ is the prior measure over the members of the partition $\mathcal{H}$,
2. $\mathbb{P}(D|H)$ is measure of $D$ relative to the subset $H$.

Note that a particular observation $D \in \mathcal{D}$ can be regarded as a subset $\{(D, H): H \in \mathcal{H} \}$ containing all the pairs consistent with $D$. This way of reasoning can easily be generalized to multiple observations. With this in mind, your answers are:

1) No, your hypotheses do not partition the set of observations.

2) Yes, the distribution of $D$ is independent of your model about the world represented by the sample space (, the $\sigma$-algebra) and the probability measure. Bayesian probability theory is just a model of how to systematically update your beliefs given your prior assumptions plus data.

-
Thank you Pedro. Could we make a concrete toy example? Let's say in reality D is distributed with N(m,v) and I have two hypothesis N(m,v) and N(m',v) with equal probability. What will be omega? –  Cagdas Ozgenc Feb 20 '13 at 16:14
In this case you have two hypotheses: $H_1 = m_1$ and $H_2 = m_2$ (note that $v$ is irrelevant). Your data space is $\mathcal{D} = \mathbb{R}$. Hence, the sample space is $\Omega = \mathcal{D} \times \mathcal{H} = \{ (x,m) : x \in \mathbb{R}, m \in \{m_1, m_2\} \}$. –  Pedro A. Ortega Feb 20 '13 at 16:19
Thanks. That made great sense now. In this case I think it is ok to say that choice of an H is partitioning omega, isn't it? The odd situation is that the total probability has nothing to do with real D. –  Cagdas Ozgenc Feb 20 '13 at 16:34
Yes, it is OK to say that a choice of an $H$ is partitioning the sample space. But no, it doesn't have anything to do with the real $D$, but that's ok, since you don't know the real $D$. What matters is that there should be at least one of the hypotheses that matches the real $D$. –  Pedro A. Ortega Feb 20 '13 at 16:37
One follow up question. If I have sequential observations, can I make a sequential update in the following manner: calculate P(H|D) and plug it back to the place of P(H) for next observation. Or do I have to make a bigger cartesian space as I observe more data and calculate P(DxD...|H)? –  Cagdas Ozgenc Feb 20 '13 at 16:42

Starting with your comment in part 2: $P(D|H)$ is not to be read as "the probability of getting this data given that I select hypothesis H". It is saying "probability of getting this given that hypothesis H is true". Clearly therefore $P(D|H) + P(D|¬H) = P(D)$

In other words, the partition can always in principle be between H is true and H is not true. By the law of the excluded middle, this must cover everything.

However, you are right to observe that if you are summing up different models that could explain the data, then there is no guarantee (or even reason to believe) that you have got "everything" covered (i.e. in practice it is usually impossible to know $P(D|¬H)$). That is why typically Bayesian analysis doesn't try to talk about the absolute probability of the model being correct given the data but only the relative probabilities of competing models.

So although I can in theory I can state:

$P(H_1 | D) = \frac{P(D|H_1) P(H_1)}{P(D)}$

$P(H_2 | D) = \frac{P(D|H_2) P(H_2)}{P(D)}$

When I do my analsis there is an implicit conditioning variable that is $M =$ "One of my models is correct". Perhaps being strict one should write:

$P(H_1 | D, M) = \frac{P(D|H_1) P(H_1|M)}{P(D|M)}$

$P(H_2 | D, M) = \frac{P(D|H_2) P(H_2|M)}{P(D|M)}$

Noting that $P(H) = P(H,M)$ since if H is true then M is true.

Now, however, notice that defined like this my priors must logically add up to 1. Often people don't have priors that add up to one. We could incorporate this by saying $P(H_1) = P(H_1|M)P(M)$. Writing our priors like this would then give us:

$P(H_1 | D, M) = \frac{P(D|H_1) P(H_1)}{P(D|M)P(M)}$

$P(H_2 | D, M) = \frac{P(D|H_2) P(H_2)}{P(D|M)P(M)}$

Of course none of this makes the slightest difference, because we are only going to compare the models, so we calculate:

$\frac{P(H_1 | D, M)}{P(H_2 | D, M)} = \frac{P(D|H_1,M) P(H_1)}{P(D|M)P(M)} \cdot \frac{P(D|M)P(M)}{P(D|H_2,M) P(H_2)} = \frac{P(D|H_1) P(H_1)} {P(D|H_2) P(H_2)}$

And all that intricacy cancels out and we are back at the original expressions.

Is it just laziness not writing the extra conditional variables that exist? Not really no. To me this is just another facet of the conditionality principle.

Edit

To take your example, suppose that $H_1$ is that $D \sim N(m,v)$ and $H_2$ is that $D \sim N(m', v)$ then clearly these two do not partition the space fully. Since we cannot evaluate $P(D|¬H_1 \cup H_2)$ we cannot work out the true $P(D)$ and so cannot ever get the absolute probability that either model is correct.

What we can do though is establish the ratio of those probabilities, since the unpartitioned part of the space cancels out (see above). This could then tell us that say, that $P(H_1|D)$ is 10 times more likely than $P(H_2|D)$.

In order to derive this, we had to write down expressions involving $P(D)$, even though we knew we'd never actually be able to calculate them.

-
"Clearly therefore P(D|H)+P(D|¬H)=P(D)". I don't agree with this statement. If my hypothesis is D~N(m',v), then probability of data given my hypothesis is true is P(D|H) = N(m',v). In that case what is P(D|not H)? 1-N(m',v)? How is that going to add up to N(m,v) which is real P(D). –  Cagdas Ozgenc Feb 20 '13 at 15:51
That is my point, it is impossible to know "not H". But you can still write down an expression for it. $P(D)$ cannot usually be calculated, unless you happen to have a situation where "H" and "not H" are known and perfectly partition the universe. If your hypothesis is $D\sim N(m,v)$ then you clearly can't define the probability of the data NOT coming from the model - it could be anything! –  Corone Feb 20 '13 at 16:04
Bayesian inference is all about chosing the most likely model, given the data. Not about evaluating the absolute probability that the model is correct. Remember "all models are wrong; some are useful". In reality $P(H)=0$, but what we really mean is not "H is true" but "H is close enough". –  Corone Feb 20 '13 at 16:07
@CagdasOzgenc I've added a bit based on your example that might make things a little clearer? –  Corone Feb 20 '13 at 16:21