Procedure to sample from an integrated marginal

Question

I would like to sample from: $$ p(\theta_2|x)=\int p(\theta_2|\theta_1,x) . p(\theta_1|x) . d\theta_1 $$

Several persons suggest me to use the following procedure to draw $(\theta_2^i)_{i=1:N}$:

for i=1:N
    draw $\theta_1^i$ from $p(\theta_1|x)$
    draw $\theta_2^i$ from $p(\theta_2|\theta_1^i,x)$

this appears intuitively satifying but I do not see how to show that this procedure is valid. It appears to be linked to Rao-Blackwellisation but it is not clear to me. Thanks for your help.

(this question is linked to an older one that I posted Sampling from marginal using integrated conditional)

Answer 1

No need to invoke Rao-Blackwellization; it's just straight probability.

Let's take conditioning on $x$ everywhere as given.

You know $p(\theta_2,\theta_1) = p(\theta_2|\theta_1)\cdot p(\theta_1)$; the sampling scheme just samples from the LHS by sampling from the scheme on the RHS - sample from $p(\theta_1)$ and then from $p(\theta_2|\theta_1)$.

When you do the first step, $\theta_1$ has the density $p(\theta_1)$. When you do the second step, conditionally, $\theta_2$ is from $p(\theta_2|\theta_1)$, but since $\theta_1$ is from its marginal, repeated sampling of the $(\theta_1,\theta_2)$ pairs is from the joint distribution, and hence the $\theta_2$ values considered alone are from their margin.