1
$\begingroup$

I am looking for document clustering approaches which gives high recall. I tried looking at Google but all I get is TF-IDF and K-means. Are there more sophisticated approaches than that which achieve a high recall?

Thanks!

Edit as per suggestions:

I have tried employing LDA using gensim. The results are rather bad. I removed the stop words, did stemming and also removed all punctuation. However when I try to infer a topic in the test, the most probable topic of each document is almost the same (>98% of the times). I have tried with 50, 100, 200, 300 and 400 topics, all give same results.

Attached is the distribution of 200(orange) topics and 300(blue) topics. (Sorry about the wrong title.)

enter image description here

enter image description here

Also, attached is the visualization of topics by pyLDAVis for 200 topics. enter image description here

Improve this question
$\endgroup$
9
  • 1
    $\begingroup$ What do you mean "high recall" when discussing document clustering? Recall generally means that you found all the items that would be deemed relevant to a search. The other concept is "precision" which is a measure of how pure your results are (not contaminated with irrelevant results). How would you define these in a clustering context? $\endgroup$
    – user75138
    Oct 12, 2016 at 20:41
  • $\begingroup$ By high recall, I mean -- if I have 4 types of documents, I expect 4 clusters and when I give a new document, it should go in the correct cluster. Sounds good enough? $\endgroup$
    – silent_dev
    Oct 13, 2016 at 5:50
  • $\begingroup$ As I said in my comments. LDA models documents as mixtures of topics, so you shouldn't infer to the most probably. For example, if you have a bunch of documents about football, don't be surprised if they all have very high weights for the topic "football, team, NFL..". You'll need to treat each document as a topic vector. $\endgroup$
    – user75138
    Oct 13, 2016 at 13:39
  • $\begingroup$ So, I did the analysis as you asked and I extracted top 10 topics for each document in the test set - it still pretty much follows the same path i.e. the Zips law. $\endgroup$
    – silent_dev
    Oct 13, 2016 at 13:43
  • $\begingroup$ I really can't help much more unless you post some results. Are you saying each document has the same topic distribution? $\endgroup$
    – user75138
    Oct 13, 2016 at 13:48

1 Answer 1

Reset to default
1
$\begingroup$

The problem with TF-IDF for clustering is that you will be working in a very high dimensional space. There's a phenomenon called the curse of dimensionality whereby methods that work in low dimensions fall apart in higher dimensions. This is particularly true of high-dimensional clustering like you find in text analysis.

In general, you have a couple choices (as outlined in the wiki article). A great primer for your case is Chapter 4 of a textbook by Charu Aggarwal (thankfully provided free by the author here) that covers text clustering.

I've used Latent Dirichlet Allocation (LDA) to reduce the dimensionality to topics (as opposed to words). Then you can apply K-means to the "topic vectors", which will often be much lower dimensional than the vocabulary space.

An addition, you should consider using a fractional distance metric as opposed to the standard Euclidean Metric. Aggarwal and his collaborators published a nice paper on this here. In many cases, the blind use of the $L_2$ norm will lead to poor classification accuracy.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks for the answer but I already tried LDA and the results are kind of disappointing and all the documents end up being in the same topic itself :( $\endgroup$
    – silent_dev
    Oct 13, 2016 at 13:07
  • $\begingroup$ @user3667569 you didn't mention that in your post...I suggest you edit to fully describe what you've done so far and where you're stuck. Also, make sure you are following good text mining practice and removing stop words and high-frequency and super low frequency words. You should not be getting all your documents into the same topic. Anyway, take a look at the book chapter in my post. It will have a lot of options that you may not have tried. Especially consider fractional distance metrics. $\endgroup$
    – user75138
    Oct 13, 2016 at 13:12
  • $\begingroup$ @user3667569 also, note that a given document will not generally be about 1 topic, so each document is represented by a vector of topics, not just the most probable. $\endgroup$
    – user75138
    Oct 13, 2016 at 13:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.