17
$\begingroup$

When using topic modeling (Latent Dirichlet Allocation), the number of topics is an input parameter that the user need to specify.

Looks to me that we should also provide a collection of candidate topic set that Dirichlet process has to sample against? Is my understanding correct? In practice, how to setup this kind of candidate topic set?

$\endgroup$
21
$\begingroup$

As far as I know you just need to supply a number of topics and the corpus. No need to specify a candidate topic set, though one can be used, as you can see in the example starting at the bottom of page 15 of Grun and Hornik (2011).

Updated 28 Jan 14. I now do things a bit differently to the method below. See here for my current approach: https://stackoverflow.com/a/21394092/1036500

A relatively simple way to find the optimum number of topics without training data is by looping through models with different numbers of topics to find the number of topics with the maximum log likelihood, given the data. Consider this example with R

# download and install one of the two R packages for LDA, see a discussion
# of them here: http://stats.stackexchange.com/questions/24441
#
install.packages("topicmodels")
library(topicmodels)
#
# get some of the example data that's bundled with the package
#
data("AssociatedPress", package = "topicmodels")

Before going right into generating the topic model and analysing the output, we need to decide on the number of topics that the model should use. Here’s a function to loop over different topic numbers, get the log liklihood of the model for each topic number and plot it so we can pick the best one. The best number of topics is the one with the highest log likelihood value to get the example data built into the package. Here I've chosen to evaluate every model starting with 2 topics though to 100 topics (this will take some time!).

best.model <- lapply(seq(2,100, by=1), function(k){LDA(AssociatedPress[21:30,], k)})

Now we can extract the log liklihood values for each model that was generated and prepare to plot it:

best.model.logLik <- as.data.frame(as.matrix(lapply(best.model, logLik)))

best.model.logLik.df <- data.frame(topics=c(2:100), LL=as.numeric(as.matrix(best.model.logLik)))

And now make a plot to see at what number of topics the highest log likelihood appears:

library(ggplot2)
ggplot(best.model.logLik.df, aes(x=topics, y=LL)) + 
  xlab("Number of topics") + ylab("Log likelihood of the model") + 
  geom_line() + 
  theme_bw()  + 
  opts(axis.title.x = theme_text(vjust = -0.25, size = 14)) + 
  opts(axis.title.y = theme_text(size = 14, angle=90))

enter image description here

Looks like it's somewhere between 10 and 20 topics. We can inspect the data to find the exact number of topics with the highest log liklihood like so:

best.model.logLik.df[which.max(best.model.logLik.df$LL),]
# which returns
       topics        LL
12     13           -8525.234

So the result is that 13 topics give the best fit for these data. Now we can go ahead with creating the LDA model with 13 topics and investigating the model:

lda_AP <- LDA(AssociatedPress[21:30,], 13)   # generate the model with 13 topics 
get_terms(lda_AP, 5)                         # gets 5 keywords for each topic, just for a quick look
get_topics(lda_AP, 5)                        # gets 5 topic numbers per document

And so on to determine the attributes of the model.

This approach is based on:

Griffiths, T.L., and M. Steyvers 2004. Finding scientific topics. Proceedings of the National Academy of Sciences of the United States of America 101(Suppl 1):5228 –5235.

$\endgroup$
  • $\begingroup$ I updated the code for this and saved as a gist. has plot method that prints by default. devtools::source_url("https://gist.githubusercontent.com/trinker/9aba07ddb07ad5a0c411/raw/c44f31042fc0bae2551452ce1f191d70796a75f9/optimal_k") +1 nice answer. $\endgroup$ – Tyler Rinker Dec 21 '15 at 4:02
  • $\begingroup$ By increasing k in LDA you are expanding the parameter space and models with smaller k are essentially nested inside models with higher k. So LL should be ever increasing with k. What you happen to a small bump around k = 13 is probably due to VEM algorithm not converging to global maximum for complex models. You will have more luck with AIC or BIC. $\endgroup$ – VitoshKa Jan 29 '16 at 15:07
  • $\begingroup$ Hi @Ben, really useful answer. I have one question about it, when you are evaluating the model with 2-100 topics: best.model <- lapply(seq(2,100, by=1), function(k){LDA(AssociatedPress[21:30,], k)}) . Why do you select only raws 21:30 of the data? $\endgroup$ – Economist_Ayahuasca Sep 26 '16 at 13:28
  • $\begingroup$ Since it was a few years ago that I posted that answer, I can't recall exactly. But probably just to keep the computation time short! $\endgroup$ – Ben Sep 26 '16 at 19:58
  • $\begingroup$ Now there is this nice pkg for computing the optimum number of topics: cran.r-project.org/web/packages/ldatuning $\endgroup$ – Ben Feb 4 '18 at 18:22

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.