# How does Gibbs sampling end up "sampling from the posterior"? [duplicate]

Let me preface this by saying I have very little experience in statistics. I did my stats courses in undergrad and most of those pertained to things like regression, not Bayesian inference.

However, for an assignment I have to give a short, rough overview of how a Gibbs sampling algorithm works. So far as I understand it, we use Gibbs sampling because the posterior distribution we would like to draw from is intractable. We draw iteratively from the conditional distributions of form p(Ui|U-i, X), updating Ui with every draw. So far so good.

The one thing that confuses me in every text I read about it is the following formulation:

After enough iterations, the Gibbs sampler draws from the target distribution p(U|X).

What does this actually mean in layman's terms, especially as it pertains to Latent Dirichlet Allocation? Does this mean the Gibbs sampler gets to a point where the actual values it is drawing approximate the latent variables of interest sufficiently? How does it "know" this?

New contributor
Yannick is a new contributor to this site. Take care in asking for clarification, commenting, and answering. Check out our Code of Conduct.
• "Does this mean the Gibbs sampler gets to a point where the actual values it is drawing approximate the latent variables of interest sufficiently?" No, not necessarily. The posterior distribution could be way off. The Gibbs sampler is only guaranteed to sample from the posterior; how useful that is depends on your model, priors, and dataset. Commented Jul 10 at 14:19

However, algorithm requires some time to reach a stationary state, where it samples proportionally to the probability density. E.g., it is clear that, if we interrupt the algorithm after just a few iterations, the results will have to do more with the starting point that we chose than with the actual distribution. This is why it is customary to discard a few hundred or a few thousand initial iterations/steps before assuming that the rest resembles the target distribution. This is what is meant by enough iterations. How much is enough has to be explored on a case-by-case basis, although there exist some helpful techniques to determine whether the algorithm has converged, such as Gelman-Rubin statistic.