3
$\begingroup$

I have a more theoretical question about LDA (Latent Dirichet Allocation).

When doing LDA we provide number of topics ourselves. As far as I understand it tries to build topic-word-document distributions to minimize perplexity (which is why we are doing it in iterative manner).

So the question - can we fit the LDA several times with different number of topics, check the perplexity of each result and choose the number of topics which yielded minimal perplexity? Or am I misunderstanding the perplexity meaning and the algorithm itself - so the perplexity is actually not a 'fit measure' for LDA?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

Yes, in fact this is the cross validation method of finding the number of topics. But note that you should minimize the perplexity of a held-out dataset to avoid overfitting.

It's worth noting that a non-parametric extension of LDA can derive the number of topics from the data without cross validation. Implementations exist at David Blei's lab github, but at the time of this writing I haven't see HDP LDA implemented in any mainstream, open-source ML libraries.

Caveat: Hierarchal LDA is different. It finds a hierarchy of topics, whereas hierarchal Dirichlet processes let you fit a potentially infinite number of flat topics. (The dual use of "hierarchy" can be confusing.)

Improve this answer
$\endgroup$
6
  • $\begingroup$ Thank you! Any advice which library has good H-LDA implementation? I do know sklearn has nothing like it, but I am not sure which one has a good implementation $\endgroup$
    – Maksim Khaitovich
    Commented Aug 11, 2017 at 17:04
  • $\begingroup$ Sorry, I haven't seen an implementation in any of the widely used libraries. David Blei's lab has two implementations, but I haven't used them. Also, keep in mind that H-LDA is different: It finds a hierarchy of topics. Hierarchal Dirichlet processes lets you fit a potentially infinite number of flat topics. (The dual use of "hierarchy" can be confusing.) $\endgroup$
    – Sean Easter
    Commented Aug 11, 2017 at 17:18
  • $\begingroup$ Now I get it. In such case any advice on where I could read a bit more about the CV method you mentioned - just want to understand if there is anything I need to keep in mind and was there any research which demonstrates potential value of performing such topics number optimization based on perplexity? $\endgroup$
    – Maksim Khaitovich
    Commented Aug 11, 2017 at 18:47
  • $\begingroup$ Sure, but clarify for me: Are you asking why we use perplexity, or why cross validate to find the number of clusters? $\endgroup$
    – Sean Easter
    Commented Aug 11, 2017 at 18:48
  • $\begingroup$ I understand why, but I am more interested in why perplexity and what are the potential benefits from business standpoint. Right now I just try to play around with number of topics manually until I get a list of topics which seem meaningful to me. But I want try and come up with the more systematic way of doing this instead of just relying on 'gut feeling' $\endgroup$
    – Maksim Khaitovich
    Commented Aug 11, 2017 at 19:02

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.