1
$\begingroup$

I am going through the derivation for Gibbs Sampling update equations for LDA. The claim is that $\theta_{d}$ (document specific topic distribution) and $\phi_k$ the topic-word distribution can be collapsed (integrated out) and one needs to estimate the individual topic assignment to each word $z_{ij}$. While I can understand the math and see it works out, I dont understand the following:

(a) How can I identify what variables can be collapsed out? For example, suppose I design a variant of LDA (like say the author-persona topic model), how might I proceed to determine which variables can be collapsed out. Are there guidelines for doing so other than just trying different possibilities? I am suspecting it has something to with Dirichlet Priors and Multinomial (conjugate pairs). (b) I read that this collapsing is an instance of Rao Blackwellization which leads me to believe that the approach used in LDA can be generalized to other variants of LDA but I am unable to see the connection yet. For example, in Slide 104 (http://perso.telecom-paristech.fr/~cappe/Research/Talks/10-peyresq_cappe.pdf) it is mentioned: "For an exponential family model, with conjugate prior the pdf $\pi(\theta|x,y)$ is available in closed form, thus allowing for Rao Blackwellization". Any pointers on how this applies to collapsed gibbs sampling for LDA would be very useful.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.