# Topic modeling (LDA) gives different outputs

I am using Topic Modeling Tool which is based on Mallet and using latent dirichlet allocation (LDA). When I ran the tool multiple times, with the same input (a folder of 200-500 short text files), and set Topics = 10 with the default of 10 words printed per each topic, the generated 10 topics and the words in the output changes.

So am i supposed to reran it multiple times and how can I pick one? Does it matter which tool of topic modelling I use? I found a few other tools in my search including Stanford Topic Modeling Toolbox, Natural Language Toolkit, etc.

• You are probably just going to have to read the documentation here. Most likely, there is a random component as a part of the underlying analyses. – gung Sep 7 '15 at 16:51
• It's worth noting that LDA is a multi-modal problem, and so one should expect different solutions given different starting points. That being said, I'm not sure what the recommended methods for choosing one solution over another in this problem would be. – Cliff AB Sep 7 '15 at 19:06

As Cliff notes in comments, the objective function for LDA is non-convex, making it a multimodal problem. That is, you can expect any given run to be locally optimal; you cannot expect that any given run would outperform some other run from different starting points.

To choose between multiple runs, consider this from another paper on variational methods by LDA creators David Blei and Michael Jordan (emphasis mine):

Practical applications of variational methods must address initialization of the variational distribution. While the algorithm yields a bound for any starting values of the variational parameters, poor choices of initialization can lead to local maxima that yield poor bounds. We initialize the variational distribution by incrementally updating the parameters according to a random permutation of the data points. (This can be viewed as a variational version of sequential importance sampling). We run the algorithm multiple times and choose the final parameter settings that give the best bound on the marginal likelihood.

They were writing about Dirichlet process mixtures, but the principle of selecting the run that best predicts the data carries. Also consider selecting based on perplexity of a held-out test set.

In the c implementation of D. Blei hyperparameters are optimized using an Expectation maximization algorithm that is initialized by n random document seeds.

This could produce slightly different results...try to set the hyperparameters fixed and see whether you get the same results...

Regarding re-running the algorithm several times (with slightly different settings of the hyperparameters): You can measure the similarity between the topics in the different iterations in order to measure "how much" the topics differ from each others. If in 2 consecutive iterations you find 4 pairs of topics with a good similarity (~90%?) you can consider the topics consistent.

Choosing the right similarity metric is another question.... In principle you can use any similarity (or dissimilarity) function between histograms (Jeffrey divergence?), or be elengant: http://www.machinelearning.org/proceedings/icml2007/papers/105.pdf

To make you analysis reproducible, use the parameter --random-seed on the command line. Finally, you select one analysis that brings out your point (depending on your research question) best.

Note that there are some other important parameters to play with:

• The number of topics
• The stopword list
• The hyperparameters
• "select one analysis that brings out your point (depending on your research question) best." this is problematic advice. Cherry-picking results to support your hypothesis is subversion of the scientific process (nice illustration of this here). Google "p-hacking" or "replicability crisis" to read more. – Rose Hartman Jan 30 '18 at 19:24
• There aren't any p-values in topic models. – jknappen Jan 31 '18 at 12:45
• What you're suggesting, even though it doesn't involve p-values, is problematic for the same reason p-hacking is. If you select your results based on whatever matches your hypotheses, you're not actually testing your hypotheses. – Rose Hartman Jan 31 '18 at 19:06
• A topic model does not test any hypothesis. There aren't hypotheses in the formal sense in a topic model. Did you ever do a topic model? – jknappen Jan 31 '18 at 19:18
• Yes, I've worked with topic models. I was using the term hypothesis lightly, to refer more to the "research questions" you mention rather than to refer to a statistical hypothesis --- my apologies for the confusion. The point remains that if you select the set of topics that "bring out your point", you run the very serious risk of misrepresenting the evidence. In a scenario like OP describes, where the topics change substantially one run to the next, it's not really appropriate to present any single run as a representative description of the documents (which is what you're advocating for). – Rose Hartman Jan 31 '18 at 19:33