# What are some of the Document Clustering approaches with high recall?

I am looking for document clustering approaches which gives high recall. I tried looking at Google but all I get is TF-IDF and K-means. Are there more sophisticated approaches than that which achieve a high recall?

Thanks!

Edit as per suggestions:

I have tried employing LDA using gensim. The results are rather bad. I removed the stop words, did stemming and also removed all punctuation. However when I try to infer a topic in the test, the most probable topic of each document is almost the same (>98% of the times). I have tried with 50, 100, 200, 300 and 400 topics, all give same results.

Attached is the distribution of 200(orange) topics and 300(blue) topics. (Sorry about the wrong title.)

Also, attached is the visualization of topics by pyLDAVis for 200 topics.

• What do you mean "high recall" when discussing document clustering? Recall generally means that you found all the items that would be deemed relevant to a search. The other concept is "precision" which is a measure of how pure your results are (not contaminated with irrelevant results). How would you define these in a clustering context? – user75138 Oct 12 '16 at 20:41
• By high recall, I mean -- if I have 4 types of documents, I expect 4 clusters and when I give a new document, it should go in the correct cluster. Sounds good enough? – silent_dev Oct 13 '16 at 5:50
• As I said in my comments. LDA models documents as mixtures of topics, so you shouldn't infer to the most probably. For example, if you have a bunch of documents about football, don't be surprised if they all have very high weights for the topic "football, team, NFL..". You'll need to treat each document as a topic vector. – user75138 Oct 13 '16 at 13:39
• So, I did the analysis as you asked and I extracted top 10 topics for each document in the test set - it still pretty much follows the same path i.e. the Zips law. – silent_dev Oct 13 '16 at 13:43
• I really can't help much more unless you post some results. Are you saying each document has the same topic distribution? – user75138 Oct 13 '16 at 13:48

An addition, you should consider using a fractional distance metric as opposed to the standard Euclidean Metric. Aggarwal and his collaborators published a nice paper on this here. In many cases, the blind use of the $L_2$ norm will lead to poor classification accuracy.