4
$\begingroup$

Suppose I have a prior $\pi(\theta)$ and likelihood $f(x|\theta)$, where $\theta$ takes values in $\{0,1\}$. I implement the following procedure

For $i = 1, \dots, T$

  • Simulate $\theta_i \sim \pi(\theta)$
  • Simulate $x_i \sim f(x|\theta_i)$

At the end of this procedure, I am left with a collection of $T$ pairs $\{(\theta_1,x_1), \dots, (\theta_T, x_T)\}$.

Suppose that $x$ takes values in $\{0,1\}$. If I then extract samples $S_{x=1} = \{\theta_i : 1\le i\le T,\; x_i = 1 \},$ it is easy to see the elements of $S_{x=1}$ are effectively samples from the posterior $\pi(\theta|x=1)$ (and similarly for $S_{x=0}$).

Suppose now that $x$ takes values in $[0,1]$. The claim is that each $\theta_i$ is a sample from the posterior $\pi(\theta|x=x_i)$. However, unlike the discrete case where I can have multiple samples from $\pi(\theta|x=0)$ and $\pi(\theta|x=1)$, in the continuous case I will have (almost surely) exactly one sample $\theta_i$ from $\pi(\theta|x=x_i)$ since all the $x_i$ are unique.

My question is whether it is true that each $\theta_i$ can be seen as a (single) sample from $\pi(\theta|x_i)$, and under what limiting process can this argument can be justified?

Note: Clearly the subtlety (and conceptual difficulty) relates to conditioning on events of probability zero, but we can easily come up with instances where the posterior distribution $\pi(\theta|x=x_i)$ is well-defined and indeed often-used (i.e. a standard conjugate model from the exponential family).

$\endgroup$

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.