I have calculated Cohen's Kapppa in order to assess inter-rater reliability where my data were coded by two independent raters. I used the "irr" package in R with the function kappa2().
For the following item (example), the responses between the two raters have a very high percent agreement but my kappa statistic ends up as NaN.
For other items I am analyzing with kappa2(), I do get kappa values that seem to make sense - and I understand that it is very common for Cohen's Kappa to be much lower than the percent agreement (sometimes extremely so). I also understand that normally, if the kappa statistic is 0, then any agreement or disagreement between raters is due to chance alone.
Q1) But what about the case where kappa outputs as "NaN"? What does this mean?
Q2) Is there any general concern that, despite multiple possible categories (1-5 for example), raters only use 1 or 2 of the possible categories (i.e., because the data are relatively homogeneous)?
The variable being rated is categorical, with possible responses of 1, 2, 3, or 4 - obviously both raters always rated as 1, except in cases of NA [where data could not be coded])
item1.rater1<-c(1,1,NA,1,1,NA,1,1,1,1,1,1,1,1) item1.rater2<-c(1,NA,NA,1,1,NA,1,1,1,1,1,NA,NA,1) item1<-as.matrix(cbind(item1.rater1,item1.rater2)) kappa2(item1[,c(1,2)]) Cohen's Kappa for 2 Raters (Weights: unweighted) Subjects = 9 Raters = 2 Kappa = NaN z = NaN p-value = NaN