1
$\begingroup$

I used the kappa2 function from the irr-library in R, however it returned an invalid z- and p-value and a kappa value of exactly 0. I already looked it up and stumbled upon this article Cohen's Kappa using (irr) and kappa2() outputs NaN, but over there the measurements are exactly the same while in my measurements (see in code below) it is not exactly the same.

I ran this code:

rm(list=ls());
library("irr")

a = c(1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 0, 1, 1)
b = c(1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1, 1)
kappa2(cbind(a, b))

And this output:

Cohen's Kappa for 2 Raters (Weights: unweighted)

 Subjects = 36 
   Raters = 2 
    Kappa = 0 

        z = NaN 
  p-value = NaN

Does anyone have an idea what goes wrong here?

$\endgroup$
0
$\begingroup$

Cohen's Kappa is

enter image description here

where in which p0 is the observed agreement between your prediction and the actual labels. pc is the randomly expected match, which is usually the proportion of label you are trying to predict. For example, if your dataset is balanced, then by chance you expect 0.5. If one of them takes up 90%, it will be 0.9

In your case, I don't know if you mixed up the prediction and reference, the reference (b) is all 1s. So pc = 1 and you end up with Inf.

$\endgroup$
4
  • $\begingroup$ I am not sure I completely understand, as I understood it the Cohens kappa is used to calculate inter-rater reliability between equally important raters. So, should the calculation not be interely symmetric? $\endgroup$ Mar 28 '20 at 13:13
  • $\begingroup$ well.. let's put it this way, if one of your rating system gives all 1, either something is rong with the rating, or you should not be using this cohen's kappa at all. $\endgroup$
    – StupidWolf
    Mar 28 '20 at 21:11
  • $\begingroup$ I deleted my other comments because in hindsight they were kinda stupid. Thanks a lot, I will look if I did something wrong when in processing my results. Otherwise, do you have any advice as which method would work better to solve this? $\endgroup$ Mar 28 '20 at 21:39
  • $\begingroup$ The data was indeed wrong. $\endgroup$ Apr 14 '20 at 18:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.