Latent Dirichlet Allocation Topic Word Distribution

Question

I trained a scikit learn LDA model on 400 000 documents of about 50 words each. I then took a look at the final topic word distribution by computing the cosine similarity matrix of the given topics. Here is the plot of this matrix:

As you can see there are 1000 topics where most of them are pretty correlated to each other but two particular topics (number 16 and 345) are not. Most words topics have probabilities around 0.0016 but if i look at these two specific topics, I get (in parenthesis are word probabilities):

cell (468.799843), protein (372.188948), using (369.602717)...

These are very common words in the corpus. Still I don't understand the meaning of such high probabilities (which sum exceeds 1). Do you understand what is happening ?

EDIT: I found out that probabilities are not normalized using lda online version. Using the following code solves this:

lda.components_ /= lda.components_.sum(axis=1)[:, np.newaxis]  

I still have the same plot though, can you interpret this ? Wrong LDA parameters ?

topic 16: cell (2.1202%), protein (1.6833%), using (1.6716%)...
topic 345: proteome (94.7656%), microfluidic (0.0058%), conservation (0.0057%)...
random topic: donor (0.1071%), editor (0.1061%), molecular (0.1061%)...

Answer 1

I happened to find this dated question. I hope you have solved it already.

I provide some tips here for people who have similar problem.

Firstly, when you are training a topic model, be sure to monitor log-likelihood. It measures how well your model fits the data.

Also, be careful when you select the number of topics. The best way to determine this parameter is by trying different values of it. Ordinarily, there is some threshold value when likelihood stops to improve even if you use larger value. In my experience, when you have about 10,000 distinct words, several hundreds of topics are enough.

Last but not least, default values for hyper-parameters are often good enough to use. If you want to tune them, I recommend you to start with very small values (e.g. 0.05).