Probability measures at play in Bayesian inference

Question

This might be a purely notational, but I'm confused about the probability measures at play when using Bayesian inference. It's sufficient to focus on the numerator here. Let's assume that I have a prior over hypotheses $P(H)$ and that these hypotheses are themselves distributions about some data. When some data $D$ is witnessed, I want to update my prior in the usual way:

$P(H|D) \propto P(D|H) P(H)$. This might be silly, but what I'm confused about is the distribution of the first term, $P(D|H)$. If I'm not completely off track, $P(D|H)$ is $H(E)$, i.e., the probability of the witnessed data under the hypothesis. However, $P(\cdot)$ is a distribution over $H$ and not over $D$, which is why $P(E|H)$ is not making much sense to me.

Where is my thinking wrong? That is, why is $P(E|H)$ defined if $P(\cdot)$ is a distribution over $H$? I'd appreciate any clarifications about the conceptual underpinnings that allow us to go from $P(E|H)$ to $P(H)$ using the same $P(\cdot)$.

--

As requested, an example that fixes the notation: Let $h \in H$ be a distribution over coin toss outcomes, and $d_{heads} \in D$ is the event of a coin landing heads. For simplicity, let's assume we only have two hypothesized distributions, $h_1$ and $h_2$, where $h_1(d_{heads}) = .5$ (fair coin), and $h_2(d_{heads}) = .25 (unfair coin).

I want to update my prior over $H$, $P(h_i)$, after witnessing a coin toss that resulted in $d_{heads}$, and apply Bayes' rule. For $h_1$ this is,

$P(h_1|d_{heads}) \propto P(d_{heads}|h_1) P(h_1)$

So, $P(h_1)$ is not an issue, as it is given by my prior. $P(d_{heads}|h_1)$ is what confuses me. This is the probability of the the coin landing heads given that $h_1$ is true -- but the probability measure $P(\cdot)$ is defined over distributions of coin tosses, not over coin tosses. That's what I'm having trouble with. Either $P(h_i)$ makes sense as the probability of hypothesis $i$, or it makes sense as $P(d_j|h_i)$ where it gives the probability of data $j$ given $i$'s truth. But I don't see how it can be both nor how to (conceptually) justify that $P(d_j|h_i)$ is often $h_i(d_j)$.

Answer 1

First, note that a Bayesian model is a joint probability distribution of all unknowns modeled as random variables. Or, if you use Bayes rule in other contexts, it still requires all random variables / events to be defined in the same joint probability space. The $P$s or $p$s then refer to marginal and conditional probabilities/distributions obtained from this joint space.

The issue may be just notational. In an attempt to clarify the notational issues, I first go over your discrete example, somewhat rigorously, considering everything as events. Second, I then point out notational problems in the more general case (where we are dealing with random variables and densities rather than events), which may have confused you since the notation annoyingly refers to different functions with the same letter $p$.

The event case (your example)

A possible probability space modeling your example situation would be the following: let the sample space be \begin{equation} \Omega = \{hh_1 , hh_2\} \times \{dd_{heads}, dd_{tails}\}, \end{equation} (I just doubled the letters to notationally separate the outcomes and the events). Let the event space be $2^\Omega$. Now, introduce notation for the events: \begin{align} h_i &= \{hh_i\} \times \{dd_{heads}, dd_{tails}\} \\ d_{foo} &= \{hh_1,hh_2\} \times \{dd_{foo}\} \end{align} There is just one probability $P$.

The point is that after specifying the following $P$-probabilities: 1. the probability of event $h_1$, 2. the conditional probability of $d_{heads}$ conditional on $h_1$, and 3. the conditional probability of $d_{heads}$ conditional on $h_2$, $P$ is uniquely determined and you may thus deduce the conditional probability of $h_1$ conditional on $d_{heads}$, using the Bayes rule. All in this same probability space.

Note also that your \begin{equation} P(h_1 \mid d_{heads}) \propto P(d_{heads} \mid h_1)\,P(h_1) \end{equation} does not make sense since both the LHS and the RHS are just single numbers, so the proportionality does not mean anything. You meant \begin{equation} P(h_i \mid d_{heads}) \propto P(d_{heads} \mid h_i)\,P(h_i), \end{equation} where the point is that the proportionality factor is the same with $i=1$ and $i=2$.

A note on notation with densities

There is a popular simplifying-but-confusing notational convention in Bayesian statistics is to just use $p(x)$ and $p(y)$ to refer to density functions (or probability mass functions in the discrete case) of $x$ and $y$ respectively. A more precise notation would i) separate the random variables and their values (say, $X$ vs. $x$) and ii) separate the different density functions by, e.g., subscripts so that $p_X(x)$ is the value of the density function of $X$ evaluated at $x$. In the popular notation, which function each $p$ refers to is left implicit. So, instead of writing

\begin{equation} p(\theta\mid y) \propto p(\theta)\,p(y\mid \theta) \end{equation}

it may be clearer to write

\begin{equation} p_{\Theta \mid Y}(\theta \mid y) \propto p_\Theta(\theta)\,p_{Y \mid \Theta}(y \mid \theta). \end{equation}