2
$\begingroup$

I would like to select the optimal number of LDA topics using R's mallet library. I know that there are several ways to do this using other implementations of LDA in R, especially using topicmodels, which allows you to maximize perplexity, maximize the harmonic mean, or maximize the negative log likelihood (though apparently this last method is not quite as good, according to the author of the post I linked to). However, I'm not sure which method works best using mallet.

I further understand that mallet protects against setting the number of topics too high by using asymmetric priors, which would seem to give some "wiggle room" with regard to selecting an ideal number of topics. However, I'm not sure how many topics is reasonable - I've experimented with everything between 20 and 200 topics (over a corpus of some 140 documents) - so this isn't much help.

How can I select the optimal number of topics?

Edit: I've tried the following code (taken from 2), and tried it out. However, it either gives what I believe is an rJava error (which can be fixed by install.packages("rJava"), at least temporarily), or it returns a list of NULL (with length equal to the length of the sequence). I'm not sure if there's a good way around this, or if going another route is better.

library(mallet)
burnin = 50
iter = 200
keep = 50
opt = 20

stopwords <- "~/path/to/stopwords.txt"
mallet.instances <- mallet.import(dataframe$id, dataframe$text, stopwords)

best.model <- lapply(seq(2,5, by=1), function(k){
  topic.model <- MalletLDA(num.topics=k)
  topic.model$loadDocuments(mallet.instances)
  topic.model$setAlphaOptimization(opt, burnin)
  topic.model$train(iter)
  })
best.model.logLik <- as.data.frame(as.matrix(lapply(best.model, logLik)))
best.model.logLik.df <- data.frame(topics=c(2:5), LL=as.numeric(as.matrix(best.model.logLik)))

library(ggplot2)
ggplot(best.model.logLik.df, aes(x=topics, y=LL)) + 
  xlab("Number of topics") + ylab("Log likelihood of the model") + 
  geom_line() + 
  theme_bw() 
$\endgroup$
  • $\begingroup$ Old question but I stumbled across this and noticed your lapply function does not return anything. Add return(topic.model) after the line topic.model$train(iter) to get the different models in a list. $\endgroup$ – JBGruber Apr 3 at 17:38
3
$\begingroup$

The best "algorithm" for selecting the number of topics is human judgement. Usually, you want to see some level of granularity in the topics and don't want to deal with superfluous detail nor a too coarsely grained version.

Thus, experiment with the number of topics unless you are satisfied with the result. Test the robustness of your topics (e.g., by varying the random seed).

$\endgroup$
  • $\begingroup$ This makes sense - however, I'm trying to use LDA to show that a particular theme (expressed by one or a few topics) appears in the corpus, and I'm worried about influencing the result by increasing the number of topics until I see the theme expressed. Is there any way that I might avoid this problem? $\endgroup$ – mlinegar Aug 25 '15 at 20:55
  • $\begingroup$ @mlinegar I won't worry to much about influencing the result by varying the number of topics. If a topic is not present in the corpus, not even a ridiculously high number of topics will bring it out. Compare the number of to topics to the magnification of a microscope: You have to choose it sensibly to see what you want to see. $\endgroup$ – jknappen Aug 26 '15 at 9:42
  • $\begingroup$ that's an interesting way to think about it - definitely helps with intuition. Are there any papers/other resources you know of that I could look to for more information on this topic, or is it seen as being more or less common sense? $\endgroup$ – mlinegar Aug 26 '15 at 21:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.