I trained a scikit learn LDA model on 400 000 documents of about 50 words each. I then took a look at the final topic word distribution by computing the cosine similarity matrix of the given topics. Here is the plot of this matrix:
As you can see there are 1000 topics where most of them are pretty correlated to each other but two particular topics (number 16 and 345) are not. Most words topics have probabilities around 0.0016 but if i look at these two specific topics, I get (in parenthesis are word probabilities):
cell (468.799843), protein (372.188948), using (369.602717)...
These are very common words in the corpus. Still I don't understand the meaning of such high probabilities (which sum exceeds 1). Do you understand what is happening ?
EDIT: I found out that probabilities are not normalized using lda online version. Using the following code solves this:
lda.components_ /= lda.components_.sum(axis=1)[:, np.newaxis]
I still have the same plot though, can you interpret this ? Wrong LDA parameters ?
topic 16: cell (2.1202%), protein (1.6833%), using (1.6716%)... topic 345: proteome (94.7656%), microfluidic (0.0058%), conservation (0.0057%)... random topic: donor (0.1071%), editor (0.1061%), molecular (0.1061%)...