Suppose I have a prior $\pi(\theta)$ and likelihood $f(x|\theta)$, where $\theta$ takes values in $\{0,1\}$. I implement the following procedure

For $i = 1, \dots, T$

  • Simulate $\theta_i \sim \pi(\theta)$
  • Simulate $x_i \sim f(x|\theta_i)$

At the end of this procedure, I am left with a collection of $T$ pairs $\{(\theta_1,x_1), \dots, (\theta_T, x_T)\}$.

Suppose that $x$ takes values in $\{0,1\}$. If I then extract samples $S_{x=1} = \{\theta_i : 1\le i\le T,\; x_i = 1 \},$ it is easy to see the elements of $S_{x=1}$ are effectively samples from the posterior $\pi(\theta|x=1)$ (and similarly for $S_{x=0}$).

Suppose now that $x$ takes values in $[0,1]$. The claim is that each $\theta_i$ is a sample from the posterior $\pi(\theta|x=x_i)$. However, unlike the discrete case where I can have multiple samples from $\pi(\theta|x=0)$ and $\pi(\theta|x=1)$, in the continuous case I will have (almost surely) exactly one sample $\theta_i$ from $\pi(\theta|x=x_i)$ since all the $x_i$ are unique.

My question is whether it is true that each $\theta_i$ can be seen as a (single) sample from $\pi(\theta|x_i)$, and under what limiting process can this argument can be justified?

Note: Clearly the subtlety (and conceptual difficulty) relates to conditioning on events of probability zero, but we can easily come up with instances where the posterior distribution $\pi(\theta|x=x_i)$ is well-defined and indeed often-used (i.e. a standard conjugate model from the exponential family).



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.