0
$\begingroup$

I am trying to understand the inference procedure of collapsed Gibbs sampling for LDA model. I refer to this document and LDA wiki page. I cannot figure out how does it simplify the sample equation especially the last two rows of the following equations to get the final result: enter image description here

Which part of them can be dropped and why?
The inference procedure in the wiki seems slightly different, but how can they get the same result?

$\endgroup$
0
$\begingroup$

This is a common trick. You want to sample, let say, a variable $z_1$ that is either 0 or 1, and you stumble on something intractable with your variable inside Gamma functions: $$ \Gamma(\alpha + \sum_n z_n) $$ You would like to save your variable and take it out of its Gamma jail, so that whatever remains on the Gamma can be treated as a constant.

So the trick is

$$ \Gamma(\alpha + \sum_n z_n) \\= \Gamma([\alpha + \sum_n z_n - z_1] + z_1)\\= \Gamma([\alpha + \sum_n z_n - z_1])(\alpha + \sum_n z_n - z_1)^{z_1}\\ \propto (\alpha + \sum_n z_n - z_1)^{z_1} $$

where we used the identity $\Gamma(a + 1) = \Gamma(a)a$.

$\endgroup$
2
  • $\begingroup$ Thank you very much! I see how they move the current sample variable out from the Gamma jail, but how do they eliminate the normalized denominator in the first term? $\endgroup$ – Sin.Sun Oct 15 '19 at 5:45
  • $\begingroup$ Once you do the Gamma trick, everything else either cancels out or is constant with respect to the variable you are sampling. The dropped denominator is just the sum over the k possible values of the numerator. It gives the number of words in document d minus one plus the sum of alphas. $\endgroup$ – alberto Oct 15 '19 at 8:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.