I am an electrical engineer and not a pro statistician.

I have a classification problem and want to compare two classifiers using Cohence kappa metric.

To calculate the kappa value, the predicted class labels (for each classifier) and the real labels are taken. So each classifier has an independent(!) kappa. Given:

method1: 0.73 (0.719 - 0.734)

method2: 0.75 (0.742 - 0.756)

I also calculate 95% CI with analytical formulas of kappa [1] wiritten in the parentheses that shows significant superiority of method2. However, the reviewers of my paper are still insisting on using statistical hypothesis test.


I could not find any clear hypothesis test for kappa.

I appreciate if someone helps me (with something like bootstrapping or sharing papers or some ideas).

[1]. J. L. Fleiss, B. Levin, and M. C. Paik, “Statistical methods for rates and proportions,” in Statistical Methods for Rates and Proportions, John Wiley & Sons, Inc., 2003, pp. 598–626

  • $\begingroup$ Please give a full citation for Fliss (not just a link, as those can rot). $\endgroup$
    – Dave
    Dec 15 '20 at 12:20

If you have large samples (which given the very narrow confidence intervals you clearly do) then using the standard errors ($s_i$) you already have you can assume $\kappa$ is normal and compute

$$ z = \frac{\kappa_1 - \kappa_2}{\sqrt{s_1^2 + s_2^2}} $$

and $z$ is a normal deviate.

  • $\begingroup$ Do you have a reference or a derivation you could include? $\endgroup$
    – Dave
    Dec 15 '20 at 13:16
  • $\begingroup$ @Dave well I suppose the OP will find it in the same chapter as they found the formula for the confidence interval. $\endgroup$
    – mdewey
    Dec 15 '20 at 14:53
  • $\begingroup$ hmmm, I got it. Indeed I have a large sample size. So I can now report the z-test result for the pair of (κ, s) for each of the classifiers and that of my reference classifier. Thanks a lot. $\endgroup$
    – Nina
    Dec 16 '20 at 9:42
  • $\begingroup$ Note that the formula assumes the samples are independent which from your comment may not be the case. I suppose you can only do what the reviewers ask but with the sample sizes you have almost any difference of scientific significance is going to be statistically significant well beyond any conventional level so what they are asking is an ignorant and stupid request but best not to say that. $\endgroup$
    – mdewey
    Dec 16 '20 at 12:26

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.