4
$\begingroup$

In the seminal paper by Tanner and Wong (1987) on data augmentation, they describe a method for obtaining the posterior distribution $p(\theta|y)$ by data augmentation. Let $Z$ be a latent variable (which will be the augmenting data) so that the posterior distribution $p(\theta|z,y)$ is tractable. Suppose for simplicity $\theta$ and $Z$ are both univariate.

At a high level, my understanding of the process described in Tanner and Wong is the following (please correct if not right): $$p(\theta|y)=\int p(\theta|z,y)p(z|y)dz$$ $$p(z|y)= \int_\Theta p(z|\phi,y)p(\phi|y)d\phi$$ up to some constant and assuming conditions that allow the exchange of integrals, we see that
\begin{equation} p(\theta|y)=\int p(\theta|z,y) \int p(z|\phi,y)p(\phi|y)d\phi dz=\int K(\theta,\phi)p(\phi|y)d\phi \hspace{.2cm} (1) \end{equation} $% it's not show equation number$ Under mild conditions, this establishes a recursion, $$p_{i+1}= \int K p_{i} d\phi$$ whose stationary state is the true distribution $p(\theta|y)$. So you need just run this long enough.

Now in middle term of (1) we see what we need to do. Obtain draws of $z|y$ using draws $\phi^{(s)}$ from $p_{i}(\phi|y)$ and then draws $z^{(s)}$ from $p(z|\phi^{(s)},y)$ (as one would with any posterior predictive distribution that is unobtainable analytically) and then approximate $p_{i+1}(\theta|y)$ using Monte Carlo integration (i.e. a mixture of conditional posteriors). Repeat until $i$ large enough.

This method relies on knowing or being able to draw from $p(\theta|z,y)$ and $p(z|\theta,y)$. So why not just use Gibbs sampling and work with draws of $\theta$ and disregard draws of $z$?

Does Gibbs not work or is it because they serve a different purpose? That is, the Tanner and Wong method gives a mixture distribution which is eventually an approximate of the true posterior $p(\theta|y)$ from which you can work as opposed to Gibbs which just gives you draws from the desired posterior?

$\endgroup$

1 Answer 1

6
$\begingroup$

As noted p. 530 in the Tanner & Wong (1987) paper, the method applies to $$p_i(\theta|y) = m^{-1}\sum_{j=1}^m p(\theta|z^{(j)},y)$$ even when $m=1$. In this special case, the algorithm is a Gibbs sampler. The algorithm remains valid in converging to the stationary distribution with the iterations for larger values of $m$, for exactly the same reason, hence it is not more or less approximate than the Gibbs sampler, but the additional cost (171mn for 15 iterations!) offers no clear advantage. Even the "final" approximation of the stationary distribution can be obtained by It is somewhat ironical that the authors dismiss the Markov connection in Remark 4 at the bottom of page 538.

$\endgroup$
9
  • $\begingroup$ Thanks @Xi'an! I had this same thought, but does it matter that the z is technically drawn from the posterior predictive z|y instead of the conditional posterior z|theta,y? $\endgroup$
    – Winston
    Commented Apr 18, 2023 at 20:52
  • 1
    $\begingroup$ But each $z^{(i)}$ is also drawn from a $p(z|\theta^{(k)},y)$ as well. so if you follow a path of pairs $(\theta,z)$ it is a Gibbs sampling path. $\endgroup$
    – Xi'an
    Commented Apr 19, 2023 at 4:54
  • $\begingroup$ There are many models in which the likelihood is better behaved with 𝑧. E.g., Chib and Greenberg 1998. One can estimate the presented model without using latent variables but the estimator would be computationally more expensive and the run times (number of samples) would be longer because the likelihood, and therefore the posterior, is "more" non-linear without 𝑧. $\endgroup$ Commented Apr 19, 2023 at 10:31
  • $\begingroup$ The computational cost of including 𝑧 is much less than the computational cost of evaluating the likelihood without 𝑧. In general MH routines often seek better behaved and computationally cheaper approaches to getting the same result. These routines may not be useful in distant future when computers are 1000x faster but are critical to practice today. $\endgroup$ Commented Apr 19, 2023 at 10:31
  • 1
    $\begingroup$ @AnirbanMukherjee the purpose of the question is how it’s described by Xi’an. I write we augment by z to work with a more tractable posterior and in particular if the Tanner method relies on knowing how to draw from the conditional posteriors (treating the latent variable as another “parameter”)…why wouldn’t one just use Gibbs instead of imputing multiple z’s. $\endgroup$
    – Winston
    Commented Apr 19, 2023 at 11:11

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.