Question: Is it reasonable to normalize topics to eliminate junk-topics and get a better distinction of document-relations?

I used the MALLET-LDA Java-library to estimate a ParallelTopicModel with 4000 iterations and K=50 topics over 2050 Files. Then I calculated the cosine similarity for all documents and exported the resulting matrix as the following graphic. Saturation is proportional to cosine-similarity and the color relates to the most relevant topic connecting both documents.

enter image description here

As you can see, the purple topic spreads over the whole text-corpus. After normalizing all topics to 1 and recalculating the document-similarities, the contrast gets higher while the relevant topics (color) stay mostly the same.

enter image description here

The refutation of this approach is that not every topic is equally present. In the case of document-exploration / recommendation the elimination of junk topics seems more relevant than the exact quantity of every topic.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.