I fit an LDA topic model, using the R package topicmodels. No hiccups and everything runs smoothly, my question here is conceptual. When controlling the Gibbs sampler, the default value (in the package topicmodels) of thinning is set to be equal to the number of iterations (and the burn-in default is set to 0).

From the package documentation we have that..: "iter, burnin, thin: These parameters control how many Gibbs sampling draws are made. The first burnin iterations are discarded and then every thin iteration is returned for iter iterations."

If I set the thinning parameter to 1, sampling indeed takes noticeable longer time even with a small dataset, so (as has been pointed out by many, e.g. here, and here, the main reason to thin the sample is to save time and memory, not to avoid autocorrelation). So far, I get it.

But what I can't seem to wrap my head around is this: in my world, this means that if we use the default setting of the topicmodels package and do 2000 iterations and set thinning ('thin') to 2000 as well and burn-in to 0, we only use the information of one (the last) sample/iteration. Then what are the use of all the other 1999 samples/iterations? Why not set 'burn-in' to 1999 and 'iter' to 1 and 'thin' to 1?

And how can a setting like thin=iterations work at all, i.e. how can keeping just one sample give us enough information to estimate the distribution?

Put differently: When we say that we only use the information from every thin iteration, that is not entirely correct is it? My understanding (and please correct me if I'm way off) is that we indeed use every iteration, even the ones between every thin, we just don't save them, otherwise, we would not have to do them at all?

I have struggled to find some useful rules of thumbs for the Gibbs sampler to lean against when doing topic modeling. For most parameters (e.g. alpha and beta), there are pretty coherent recommendations with explanations out there, but not for thinning, except possibly here, p. 171:

"In practice, we find it is generally more than enough to save 1000 iterations in total, and so we thin accordingly." That would mean, with e.g. 2000 iterations (after discarding x burn-in), to set thinning to 2, as I understand it.

However, scientists use wildly different 'thinning-practices', judging by online examples and tutorials (my own included) of topic modeling. Many use the topicmodels package without specifying thinning at all, thus effectively using (unknowingly?) the default iter=thin.

Could someone clear the fog for me? I guess I have missed something essential about what exactly is kept by every iteration and about how thinning works.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.