5
$\begingroup$

For two classifiers h1 and h2 I have the precision, recall and F1 score as a percentage (along with the original labeled data set that they were tested on). If I had access to which samples each classifier classified right/wrong, I would be able to do, for example, McNemar's test to evaluate significance, but unfortunately I don't.

I would ideally like to be able to speak on the significance of the results obtained by h2, that is, whether h2 is a significant improvement over h1. Am I unable to do that, or is there something I can say using only precision/recall/F1 and the labeled data set?

$\endgroup$
  • $\begingroup$ Do you have the quality values per class at least, or do you only have a global measure only? $\endgroup$ – miguelmalvarez Jun 4 '13 at 21:20
  • $\begingroup$ Unfortunately not. It looks like I'll have to re-train the models and get proper output - Doesn't look like I can say much based on what I'd gathered previously. $\endgroup$ – Alathon Jun 5 '13 at 13:01
1
$\begingroup$

If all you have is P/R/F1 scores for the two systems/classifiers there's no way of testing whether the difference between the two is statistically significant. For the McNemar's test, as you suggested, you would need the predictions of the two systems.

If you have other labeled data and the implementation of the two systems, you can test those on the data (shuffling and 5- or 10-fold cross-validating several times) so that you can perform a statistical test.

$\endgroup$
0
$\begingroup$

You have different options about how to compare different classifiers in the text domain. However, you will need quality levels per class or per document. You can check this paper about Re-examining text categorisation methods.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.