Popular python libraries for topic modeling like gensim or sklearn allow us to predict the topic-distribution for an unseen document, but I have a few questions on what's going on under the hood. I've read a few responses about "folding-in", but the Blei et al. LDA paper the authors state.

" An alternative approach is the “folding-in” heuristic suggested by Hofmann (1999), where one ignores the p(z|d) parameters and refits p(z|dnew). Note that this gives the pLSI model an unfair advantage by allowing it to refit k −1 parameters to the test data. LDA suffers from neither of these problems. As in pLSI, each document can exhibit a different proportion of underlying topics. However, LDA can easily assign probability to a new document; no heuristics are needed for a new document to be endowed with a different set of topic proportions than were associated with documents in the training corpus."

Which makes me thing folding-in may not be the right way to predict topics for LDA. Furthermore, I'm curious about how we could predict topic mixtures for documents with only access to the topic-word distribution $\Phi$. Essentially, I want the document-topic mixture $\theta$ so we need to estimate $p(\theta_z | d, \Phi)$ for each topic $z$ for an unseen document $d$.

I might be overthinking it. Can we sample from $\Phi$ for each word in $d$ until each $\theta_z$ converges?



Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.