1
$\begingroup$

I have created a text clustering algorithm and calculated the Precision and Recall measures for the evaluation. I am looking for papers that contain other text clustering algorithms evaluations with Precision and Recall in order to compare them.

I have already found a few from Google, but I am not satisfied with the number of them. Do you have any to suggest?

Update:

Upon request, I update my question to include more information. I used TF-IDF for similarity and hierarchical clustering.

$\endgroup$
  • $\begingroup$ This question belies the wide breadth of such algorithms in existence and the particular applications each is meant to achieve. For instance, in bioinformatics, it is desirable to search for "motifs" (frequently co-occurring basepairs) that can have any number of random insertions or deletions according to a know frequency of words. You can have supervised or unsupervised clustering applications for text analytics. I think you need to expound upon your question. $\endgroup$ – AdamO Jan 7 '14 at 19:32
  • $\begingroup$ @AdamO Thank you for your advice. I have included a few more info. TF-IDF and hierarchical clustering have been used. $\endgroup$ – Tasos Jan 7 '14 at 20:28
  • $\begingroup$ Precision and Recall are classification measures, not clustering measures. $\endgroup$ – Has QUIT--Anony-Mousse Jan 8 '14 at 13:57
1
$\begingroup$

There are two issues that make it hard to give you a concise answer.

  1. What do you want to do? The median "text miner" probably works on articles scraped from the web, but there are also people using similar tools to analyse DNA, identify malicious software (or users), and nearly anything else that involves sequences of tokens. (See also @AdamO's comment above).

  2. How do you want to evaluate it? Classification is easy: your data has some labels and just have to determine how well the algorithm has reproduced them from the rest of your data. Clustering, on the other hand, is essentially asking your code to find some sensible way to arrange your data. Since you don't have ground-truth labels, it is not clear to me how you would directly compute precision and recall from a clustering run. People sometimes use EM (or another alignment technique) to align their cluster assignments with labels, but you still need to verify that it's not clustering according to some other plausible category (author rather than topic, for example).

Furthermore, comparing precision and recall (or any other performance metric) against other papers isn't going to be very helpful unless you're also running on the same data sets. Some corpora have become de-facto standards. The RCV1 data set is one common benchmark. It's a huge collection of ~ a million news articles from Reuters, labelled with a controlled vocabulary. Lin and Cohen (2010) tested a few classification algorithms on it, so that might be one place to start. Ron Bekkerman has published some experiments using the Enron Corpus, as have several other people, so that might be a place to look if you're interested in "real" email or author identification.

There are other, more specialized data sets floating around too, ranging from movie reviews (labelled with ratings) to biomedical texts (with gene and protein names identified), depending on your interests.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.