3
$\begingroup$

I have calculated Cohen's Kapppa in order to assess inter-rater reliability where my data were coded by two independent raters. I used the "irr" package in R with the function kappa2().

For the following item (example), the responses between the two raters have a very high percent agreement but my kappa statistic ends up as NaN.

For other items I am analyzing with kappa2(), I do get kappa values that seem to make sense - and I understand that it is very common for Cohen's Kappa to be much lower than the percent agreement (sometimes extremely so). I also understand that normally, if the kappa statistic is 0, then any agreement or disagreement between raters is due to chance alone.

Q1) But what about the case where kappa outputs as "NaN"? What does this mean?

Q2) Is there any general concern that, despite multiple possible categories (1-5 for example), raters only use 1 or 2 of the possible categories (i.e., because the data are relatively homogeneous)?


The variable being rated is categorical, with possible responses of 1, 2, 3, or 4 - obviously both raters always rated as 1, except in cases of NA [where data could not be coded])

item1.rater1<-c(1,1,NA,1,1,NA,1,1,1,1,1,1,1,1)
item1.rater2<-c(1,NA,NA,1,1,NA,1,1,1,1,1,NA,NA,1)
item1<-as.matrix(cbind(item1.rater1,item1.rater2))
kappa2(item1[,c(1,2)])

Cohen's Kappa for 2 Raters (Weights: unweighted)

Subjects = 9 
  Raters = 2 
   Kappa = NaN 

       z = NaN 
 p-value = NaN 
$\endgroup$
4
$\begingroup$

In this data, your two raters agreed 100% of the time (the function excludes all rows that include an NA). They both individually have a 100% estimated chance of selecting category "1". Thus, the probability of agreement by chance is $p_e = 1*1 = 1$. Notice that the denominator of Kappa includes $1-p_e$. Thus you are trying to divide by zero. In R, since the numerator also happens to be zero, you end up with $0/0 = NaN$.

It seems to me this is an edge case that the irr package developers might want address in their next version of the package.

$\endgroup$
  • 2
    $\begingroup$ This is an issue with the formulation of kappa, not with the irr package. $\endgroup$ – Jeffrey Girard Apr 12 '16 at 12:20
  • 1
    $\begingroup$ @JeffreyGirard, yes. I don't mean to suggest that irr has an error, but only to point out an opportunity to make irr a little more user friendly with a new warning feature. $\endgroup$ – zkurtz Apr 12 '16 at 16:14
  • $\begingroup$ This is not an answer per se but an acknowledgement. I found the answer on the NaN output question above also helpful for my own study. It would be nice though if you could add a citation or a source to the answer as I am struggling to find a supporting source for the information that you have provided in your answer above. $\endgroup$ – user135547 Oct 20 '16 at 14:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.