I am asking 1 question to 7 raters. They reply with one of three possible nominal values; for the sake of this example, let's call them 1, 2, and 3.
I need a measure of inter-rater reliability that is high when people mostly agree on the answer and low otherwise.
My first thought was to use Krippendorff's alpha, but I'm getting what seems like very counter-intuitive results.
If one person says '2' and the other six all say '1' (in R):
library(irr) kripp.alpha(matrix(c(2,1,1,1,1,1,1),nrow=7)) Krippendorff's alpha Subjects = 1 Raters = 7 alpha = -0.139
It seems odd that I would be getting a negative value here. Surely 6 out of 7 people giving the same answer should represent some form of agreement? A one-tailed Binomial test tells me that the probability of this happening by chance are very low (p < 0.006).
So I thought maybe 6 out of 7 just wan't good enough, and I started exploring other values to see how many people I would need to detect agreement.
If one person says '2' and 49 other people say '1':
library(irr) kripp.alpha(matrix(c(2,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1),nrow=50)) Krippendorff's alpha Subjects = 1 Raters = 50 alpha = -0.2
How can it be that 49 out of 50 people giving the same answer represents less agreement than 6 out of 7?
I'm getting similar results for Fleiss' kappa (though I'm not sure I'm even supposed to use Fleiss's for nonimal data).
Please advise. I guess my specific questions are:
- Am I calculating these values wrongly?
- Am I misunderstanding? Does 49 out of 50 people giving the same answer really represent disagreement?
- If I'm not doing this wrongly, what statistic should I use instead? I need something that is high when people mostly give the same answer to a nominal-valued question and low otherwise.