1
$\begingroup$

Let me preface this by saying I have very little experience in statistics. I did my stats courses in undergrad and most of those pertained to things like regression, not Bayesian inference.

However, for an assignment I have to give a short, rough overview of how a Gibbs sampling algorithm works. So far as I understand it, we use Gibbs sampling because the posterior distribution we would like to draw from is intractable. We draw iteratively from the conditional distributions of form p(Ui|U-i, X), updating Ui with every draw. So far so good.

The one thing that confuses me in every text I read about it is the following formulation:

After enough iterations, the Gibbs sampler draws from the target distribution p(U|X).

What does this actually mean in layman's terms, especially as it pertains to Latent Dirichlet Allocation? Does this mean the Gibbs sampler gets to a point where the actual values it is drawing approximate the latent variables of interest sufficiently? How does it "know" this?

New contributor
Yannick is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
$\endgroup$
1
  • $\begingroup$ "Does this mean the Gibbs sampler gets to a point where the actual values it is drawing approximate the latent variables of interest sufficiently?" No, not necessarily. The posterior distribution could be way off. The Gibbs sampler is only guaranteed to sample from the posterior; how useful that is depends on your model, priors, and dataset. $\endgroup$ Commented Jul 10 at 14:19

1 Answer 1

0
$\begingroup$

Gibbs sampling and closely related Metropolis-Hastings algorithm are designed as random processes (random walks) through the variable space, which end up spending more time in the regions of higher probability density and less or no time at all in the regions of the low probability density of the target distribution. These algorithms are explicitly designed in such a way that the time they spend in each such region (which is the number of points drawn from this region) is proportional to the target probability density.

However, algorithm requires some time to reach a stationary state, where it samples proportionally to the probability density. E.g., it is clear that, if we interrupt the algorithm after just a few iterations, the results will have to do more with the starting point that we chose than with the actual distribution. This is why it is customary to discard a few hundred or a few thousand initial iterations/steps before assuming that the rest resembles the target distribution. This is what is meant by enough iterations. How much is enough has to be explored on a case-by-case basis, although there exist some helpful techniques to determine whether the algorithm has converged, such as Gelman-Rubin statistic.

$\endgroup$

Not the answer you're looking for? Browse other questions tagged or ask your own question.