# Monte Carlo Options for Data Augmentation

In the seminal paper by Tanner and Wong (1987) on data augmentation, they describe a method for obtaining the posterior distribution $$p(\theta|y)$$ by data augmentation. Let $$Z$$ be a latent variable (which will be the augmenting data) so that the posterior distribution $$p(\theta|z,y)$$ is tractable. Suppose for simplicity $$\theta$$ and $$Z$$ are both univariate.

At a high level, my understanding of the process described in Tanner and Wong is the following (please correct if not right): $$p(\theta|y)=\int p(\theta|z,y)p(z|y)dz$$ $$p(z|y)= \int_\Theta p(z|\phi,y)p(\phi|y)d\phi$$ up to some constant and assuming conditions that allow the exchange of integrals, we see that
$$$$p(\theta|y)=\int p(\theta|z,y) \int p(z|\phi,y)p(\phi|y)d\phi dz=\int K(\theta,\phi)p(\phi|y)d\phi \hspace{.2cm} (1)$$$$ $$% it's not show equation number$$ Under mild conditions, this establishes a recursion, $$p_{i+1}= \int K p_{i} d\phi$$ whose stationary state is the true distribution $$p(\theta|y)$$. So you need just run this long enough.

Now in middle term of (1) we see what we need to do. Obtain draws of $$z|y$$ using draws $$\phi^{(s)}$$ from $$p_{i}(\phi|y)$$ and then draws $$z^{(s)}$$ from $$p(z|\phi^{(s)},y)$$ (as one would with any posterior predictive distribution that is unobtainable analytically) and then approximate $$p_{i+1}(\theta|y)$$ using Monte Carlo integration (i.e. a mixture of conditional posteriors). Repeat until $$i$$ large enough.

This method relies on knowing or being able to draw from $$p(\theta|z,y)$$ and $$p(z|\theta,y)$$. So why not just use Gibbs sampling and work with draws of $$\theta$$ and disregard draws of $$z$$?

Does Gibbs not work or is it because they serve a different purpose? That is, the Tanner and Wong method gives a mixture distribution which is eventually an approximate of the true posterior $$p(\theta|y)$$ from which you can work as opposed to Gibbs which just gives you draws from the desired posterior?

As noted p. 530 in the Tanner & Wong (1987) paper, the method applies to $$p_i(\theta|y) = m^{-1}\sum_{j=1}^m p(\theta|z^{(j)},y)$$ even when $$m=1$$. In this special case, the algorithm is a Gibbs sampler. The algorithm remains valid in converging to the stationary distribution with the iterations for larger values of $$m$$, for exactly the same reason, hence it is not more or less approximate than the Gibbs sampler, but the additional cost (171mn for 15 iterations!) offers no clear advantage. Even the "final" approximation of the stationary distribution can be obtained by It is somewhat ironical that the authors dismiss the Markov connection in Remark 4 at the bottom of page 538.
• But each $z^{(i)}$ is also drawn from a $p(z|\theta^{(k)},y)$ as well. so if you follow a path of pairs $(\theta,z)$ it is a Gibbs sampling path. Apr 19, 2023 at 4:54