1
$\begingroup$

I am interested in comparing Classifier A with Classifier B. I have obtained Micro-Averaged F1 measures for Classifiers A and B that I intend to compare pairwise. I want to find out if Classifier A is better than B.

I am a little unclear on how actually to conduct the Wilcoxon Signed Rank test. As far as my understanding goes, the null hypothesis is that there is no significant difference between the classifiers, and the alternate hypothesis is that there is. Is this correct? If this is indeed true, then how do I in fact show that A is better than B - because in this case even if I fail to reject the null hypothesis, all I have shown is that there's a significant difference in classifier performance, and not that A is better than B...

$\endgroup$
1
$\begingroup$

There are many approaches, most of them not very powerful (e.g., comparing two ROC areas (c-indexes)). Two powerful approaches, most easily done in an independent validation sample, are as follows, after making sure that you get much more than information-losing "classification" out of the "classifiers". Efficient approaches need e.g. estimated probabilities of class membership.

  1. Embed the continuous predicted values from both methods into a "super model" and do a likelihood-ratio $\chi^2$ test for whether method A adds to method B, and the reverse. If one adds to the other and the reverse is not true, the one is clearly better than the other.
  2. Use the R Hmisc package rcorrp.cens to test the null hypothesis that method A is no more concordant with the outcome than method B. This approach is more powerful than testing differences in ROC areas, and it works by forming all possible pairs of pairs of predictions.
$\endgroup$
  • $\begingroup$ Thank you for your answer, but I have been asked to do the Wilcoxon signed rank test, and I am afraid I am stuck with that. What has got me in a tangle is if I have the null and alternate hypotheses correct, and what I need to do with the p-value that the software (scipy in this case) gives me. $\endgroup$ – Jimmy Jun 21 '15 at 12:28
  • 1
    $\begingroup$ Beware of doing what you've been asked to do when it is suboptimal or wrong. Among many other things, the signed rank test ignores the actual outcomes. $\endgroup$ – Frank Harrell Jun 21 '15 at 14:09
  • $\begingroup$ I sort of found a way to mitigate the damage - viz reporting the signed ranks. That will have to do for now :-(. So if I obtain a (two sided) p-value using scipy, would I be correct in rejecting the null hypothesis if this p-value is less than the significance level (say 0.05)? And would my null hypothesis be that there is no "significant" difference between the classifiers, and the alternative hypothesis being that there is? Sorry, I don't have a stats background, and just can't seem to be able to wrap my head around this. Thanks for your help and patience ! $\endgroup$ – Jimmy Jul 1 '15 at 17:12
  • $\begingroup$ This makes no sense to me, sorry. What is the exact hypothesis you think the Wilcoxon signed rank test is testing for you? That test is used to test whether there is a systematic difference in a variable, not whether there is a difference in how the variables relate to another variable (the true outcomes, for example). $\endgroup$ – Frank Harrell Jul 1 '15 at 21:10
  • $\begingroup$ Given the F1 measures of the classifiers A and B corresponding to the multiple classes (~20 in my case), the null hypothesis states that there is no discernible difference between the F1 values of the classifiers, and the alternate hypothesis that there is (for now, please disregard the alternate hypothesis "A is better than B" that I state in my original question). I am actually seeing (2-sided) p-value of 0.0002 using scipy. So since my significance level is 0.05, can I reject the null hypothesis? $\endgroup$ – Jimmy Jul 2 '15 at 7:38
1
$\begingroup$

The use of Wilcoxon signed rank test is the recommended method for comparing two classifiers in

Demšar, Janez. "Statistical comparisons of classifiers over multiple data sets." Journal of Machine learning research 7. Jan (2006): 1-30. http://www.jmlr.org/papers/volume7/demsar06a/demsar06a.pdf

This paper also provides some guidance on how to apply the test.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.