The input parameters for using latent Dirichlet allocation

Question

When using topic modeling (Latent Dirichlet Allocation), the number of topics is an input parameter that the user need to specify.

Looks to me that we should also provide a collection of candidate topic set that Dirichlet process has to sample against? Is my understanding correct? In practice, how to setup this kind of candidate topic set?

Answer 1

As far as I know you just need to supply a number of topics and the corpus. No need to specify a candidate topic set, though one can be used, as you can see in the example starting at the bottom of page 15 of Grun and Hornik (2011).

Updated 28 Jan 14. I now do things a bit differently to the method below. See here for my current approach: https://stackoverflow.com/a/21394092/1036500

A relatively simple way to find the optimum number of topics without training data is by looping through models with different numbers of topics to find the number of topics with the maximum log likelihood, given the data. Consider this example with R

# download and install one of the two R packages for LDA, see a discussion
# of them here: http://stats.stackexchange.com/questions/24441
#
install.packages("topicmodels")
library(topicmodels)
#
# get some of the example data that's bundled with the package
#
data("AssociatedPress", package = "topicmodels")

Before going right into generating the topic model and analysing the output, we need to decide on the number of topics that the model should use. Here’s a function to loop over different topic numbers, get the log liklihood of the model for each topic number and plot it so we can pick the best one. The best number of topics is the one with the highest log likelihood value to get the example data built into the package. Here I've chosen to evaluate every model starting with 2 topics though to 100 topics (this will take some time!).

best.model <- lapply(seq(2,100, by=1), function(k){LDA(AssociatedPress[21:30,], k)})

Now we can extract the log liklihood values for each model that was generated and prepare to plot it:

best.model.logLik <- as.data.frame(as.matrix(lapply(best.model, logLik)))

best.model.logLik.df <- data.frame(topics=c(2:100), LL=as.numeric(as.matrix(best.model.logLik)))

And now make a plot to see at what number of topics the highest log likelihood appears:

library(ggplot2)
ggplot(best.model.logLik.df, aes(x=topics, y=LL)) + 
  xlab("Number of topics") + ylab("Log likelihood of the model") + 
  geom_line() + 
  theme_bw()  + 
  opts(axis.title.x = theme_text(vjust = -0.25, size = 14)) + 
  opts(axis.title.y = theme_text(size = 14, angle=90))

Looks like it's somewhere between 10 and 20 topics. We can inspect the data to find the exact number of topics with the highest log liklihood like so:

best.model.logLik.df[which.max(best.model.logLik.df$LL),]
# which returns
       topics        LL
12     13           -8525.234

So the result is that 13 topics give the best fit for these data. Now we can go ahead with creating the LDA model with 13 topics and investigating the model:

lda_AP <- LDA(AssociatedPress[21:30,], 13)   # generate the model with 13 topics 
get_terms(lda_AP, 5)                         # gets 5 keywords for each topic, just for a quick look
get_topics(lda_AP, 5)                        # gets 5 topic numbers per document

And so on to determine the attributes of the model.

This approach is based on:

Griffiths, T.L., and M. Steyvers 2004. Finding scientific topics. Proceedings of the National Academy of Sciences of the United States of America 101(Suppl 1):5228 –5235.