In the collapsed Gibbs sampling version of LDA, the posterior distribution of topic assignments for each word is sampled. From what I have read (e.g. http://people.cs.umass.edu/~wallach/courses/s11/cmpsci791ss/readings/griffiths02gibbs.pdf) it looks like people just implement the Gibbs sampling procedure by letting it run until we are sure that we are sampling from the posterior then take the last sample as a good topic assignment.

But shouldn't we be a bit more rigorous than this? For example, what if the sample we take happens to be from a part of the posterior with low probability (this would be unlikely but possible)? Wouldn't it be better to do something like take a Bayes estimator (or average, or mode) of the samples to make sure that we are using a topic assignment that is actually likely?

  • $\begingroup$ This is also what GibbsLDA++ seems to do. I'm surprised that this is the favored approach since, as you said, there are severe theoretical problems. I would think it would make a lot more sense to at least choose the iteration that maximized the likelihood of the model out of the last $T$ iterations instead of just choosing the last iteration. $\endgroup$ – user44764 Jun 11 '14 at 20:29

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.