3
$\begingroup$

When increasing the value of the latent variable k for LDA (latent Dirichlet allocation), how should perplexity behave:

  1. On the training set?
  2. On the testing set?
$\endgroup$

1 Answer 1

2
$\begingroup$

The original paper on LDA gives some insights into this:

In particular, we computed the perplexity of a held-out test set to evaluate the models. The perplexity, used by convention in language modeling, is monotonically decreasing in the likelihood of the test data, and is algebraicly equivalent to the inverse of the geometric mean per-word likelihood. A lower perplexity score indicates better generalization performance.

This should be the behavior on test data. Here is a result from paper: Perplexity as a function of k

$\endgroup$
6
  • $\begingroup$ Thank you, I thought it was supposed to decrease, but wanted to confirm because scikit-learn was showing an increase on my dataset. Could you add citation details (date, journal, etc.) to the post? I assume the paper is "Latent Dirichlet Allocation" by Blei, Ng, & Jordan, but I don't see the image you are referring to in the version of the paper I am viewing. $\endgroup$
    – user179041
    Oct 14, 2017 at 16:32
  • $\begingroup$ Any idea about how it should behave on the training set? I assume perplexity should also decrease on training as k increases (while, for the same k, perplexity of training should be less than in testing) but have not been able to confirm this $\endgroup$
    – user179041
    Oct 14, 2017 at 16:40
  • $\begingroup$ @user179041 Here is the link to the paper: Since perplexity decreases with increasing likelihood of the data, I would imagine with increasing K, the perplexity to decrease. $\endgroup$
    – kedarps
    Oct 17, 2017 at 15:04
  • $\begingroup$ Thank you for the link. From looking at the paper, the image you included above (fig 9) is for performance on the document modeling task. But, there is another image (fig 11) for the collaborative filtering task performance which shows a local minimum then slight increase. However, the quote you included suggests that perplexity should be monotonically decreasing on test data. How do you think we should interpret this seemingly contradictory information? $\endgroup$
    – user179041
    Oct 19, 2017 at 16:08
  • $\begingroup$ The perplexity metric does perplex me ;). See this related question I asked earlier. I think one should not expect a decreasing perplexity with increasing K. The choice of K should be based on the value that minimizes perplexity. $\endgroup$
    – kedarps
    Oct 19, 2017 at 19:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.