0
$\begingroup$

I have clustered over 4000 textual files, and now I want to check and evaluate clusters. I want to use F-measure (a mix of recall and precision).

The formal definition of F1_score is:
$$ \text{F-measure} = 2 * \frac{\text{precision} * \text{recall}}{\text{precision} + \text{recall}} $$ To calculate the precision and recall, I have to know in advance the correct number of documents retrieved and correct not-retrieved. That means I need to browse all documents and see what they contain. But the problem is, how can I know the precision and recall of more than 4000 textual files?

Improve this question
$\endgroup$
0

1 Answer 1

Reset to default
1
$\begingroup$

It seems that you have an unlabeled data set. The F1 score requires that you compare your experiment's results with actual results (i.e., count false positives and negatives). If you don't have any "correct" results against which to compare, you won't be able to calculate a F1 score.

To calculate Precision/Recall in an unsupervised approach, you need training data just for that purpose. You could manually label some data, or find a way to generate training data algorithmically, but these are difficult things to do.

You may be better off choosing a different and more applicable method by which to "check and evaluate" your clusters.

Improve this answer
$\endgroup$
2
  • $\begingroup$ Yes , i have label data , it is the words that in document with it's frequencies. my problem how can i know and calculate recall and Precision. then calculate F1 score $\endgroup$
    – wael
    Apr 22, 2013 at 18:26
  • 2
    $\begingroup$ Term frequencies are not category labels. Your clustering algorithms is placing your data objects (documents?) into several categories. The labels you would need are mappings from those objects (documents) to the correct categories. You could then compare your clustering results to these correct categories. Without this labeled training data, you should consider a different approach entirely. $\endgroup$
    – Aman
    Apr 22, 2013 at 18:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.