I am using Topic Modeling Tool which is based on Mallet and using latent dirichlet allocation (LDA). When I ran the tool multiple times, with the same input (a folder of 200-500 short text files), and set Topics = 10 with the default of 10 words printed per each topic, the generated 10 topics and the words in the output changes.

So am i supposed to reran it multiple times and how can I pick one? Does it matter which tool of topic modelling I use? I found a few other tools in my search including Stanford Topic Modeling Toolbox, Natural Language Toolkit, etc.

  • $\begingroup$ You are probably just going to have to read the documentation here. Most likely, there is a random component as a part of the underlying analyses. $\endgroup$ – gung - Reinstate Monica Sep 7 '15 at 16:51
  • 1
    $\begingroup$ It's worth noting that LDA is a multi-modal problem, and so one should expect different solutions given different starting points. That being said, I'm not sure what the recommended methods for choosing one solution over another in this problem would be. $\endgroup$ – Cliff AB Sep 7 '15 at 19:06

As Cliff notes in comments, the objective function for LDA is non-convex, making it a multimodal problem. That is, you can expect any given run to be locally optimal; you cannot expect that any given run would outperform some other run from different starting points.

To choose between multiple runs, consider this from another paper on variational methods by LDA creators David Blei and Michael Jordan (emphasis mine):

Practical applications of variational methods must address initialization of the variational distribution. While the algorithm yields a bound for any starting values of the variational parameters, poor choices of initialization can lead to local maxima that yield poor bounds. We initialize the variational distribution by incrementally updating the parameters according to a random permutation of the data points. (This can be viewed as a variational version of sequential importance sampling). We run the algorithm multiple times and choose the final parameter settings that give the best bound on the marginal likelihood.

They were writing about Dirichlet process mixtures, but the principle of selecting the run that best predicts the data carries. Also consider selecting based on perplexity of a held-out test set.

| cite | improve this answer | |

In the c implementation of D. Blei hyperparameters are optimized using an Expectation maximization algorithm that is initialized by n random document seeds.

This could produce slightly different results...try to set the hyperparameters fixed and see whether you get the same results...

Regarding re-running the algorithm several times (with slightly different settings of the hyperparameters): You can measure the similarity between the topics in the different iterations in order to measure "how much" the topics differ from each others. If in 2 consecutive iterations you find 4 pairs of topics with a good similarity (~90%?) you can consider the topics consistent.

Choosing the right similarity metric is another question.... In principle you can use any similarity (or dissimilarity) function between histograms (Jeffrey divergence?), or be elengant: http://www.machinelearning.org/proceedings/icml2007/papers/105.pdf

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.