I am going through the derivation for Gibbs Sampling update equations for LDA. The claim is that $\theta_{d}$ (document specific topic distribution) and $\phi_k$ the topic-word distribution can be collapsed (integrated out) and one needs to estimate the individual topic assignment to each word $z_{ij}$. While I can understand the math and see it works out, I dont understand the following:

(a) How can I identify what variables can be collapsed out? For example, suppose I design a variant of LDA (like say the author-persona topic model), how might I proceed to determine which variables can be collapsed out. Are there guidelines for doing so other than just trying different possibilities? I am suspecting it has something to with Dirichlet Priors and Multinomial (conjugate pairs). (b) I read that this collapsing is an instance of Rao Blackwellization which leads me to believe that the approach used in LDA can be generalized to other variants of LDA but I am unable to see the connection yet. For example, in Slide 104 (http://perso.telecom-paristech.fr/~cappe/Research/Talks/10-peyresq_cappe.pdf) it is mentioned: "For an exponential family model, with conjugate prior the pdf $\pi(\theta|x,y)$ is available in closed form, thus allowing for Rao Blackwellization". Any pointers on how this applies to collapsed gibbs sampling for LDA would be very useful.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.