Krippendorff's alpha (and other IRR stats) giving counter-intuitive results

Question

I am asking 1 question to 7 raters. They reply with one of three possible nominal values; for the sake of this example, let's call them 1, 2, and 3.

I need a measure of inter-rater reliability that is high when people mostly agree on the answer and low otherwise.

My first thought was to use Krippendorff's alpha, but I'm getting what seems like very counter-intuitive results.

If one person says '2' and the other six all say '1' (in R):

library(irr)
kripp.alpha(matrix(c(2,1,1,1,1,1,1),nrow=7))
Krippendorff's alpha
Subjects = 1 
Raters = 7 
alpha = -0.139 

It seems odd that I would be getting a negative value here. Surely 6 out of 7 people giving the same answer should represent some form of agreement? A one-tailed Binomial test tells me that the probability of this happening by chance are very low (p < 0.006).

So I thought maybe 6 out of 7 just wan't good enough, and I started exploring other values to see how many people I would need to detect agreement.

If one person says '2' and 49 other people say '1':

library(irr)
kripp.alpha(matrix(c(2,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1,1),nrow=50))
Krippendorff's alpha
Subjects = 1 
Raters = 50 
alpha = -0.2

How can it be that 49 out of 50 people giving the same answer represents less agreement than 6 out of 7?

I'm getting similar results for Fleiss' kappa (though I'm not sure I'm even supposed to use Fleiss's for nonimal data).

Please advise. I guess my specific questions are:

• Am I calculating these values wrongly?
• Am I misunderstanding? Does 49 out of 50 people giving the same answer really represent disagreement?
• If I'm not doing this wrongly, what statistic should I use instead? I need something that is high when people mostly give the same answer to a nominal-valued question and low otherwise.

Answer 1

Chance-adjusted indexes of categorical agreement measure the amount of observed agreement and then attempt to "adjust" it using some estimate of how much agreement would be expected by chance alone (e.g., guessing). The reason there are different indexes is because they estimate chance agreement in different ways.

The problem you are describing will happen with any chance-adjusted index of categorical agreement that estimates chance agreement by considering the distribution of categories (e.g., alpha, kappa, and pi). Basically, the distribution of your categories is so skewed (i.e., 1 is so much more common than 2) that guessing category 1 would almost always be correct. As such, these indexes estimate chance agreement to be very high and "adjust" it down to be very low or even negative.

There is debate in the field about whether this is desirable behavior for an agreement index (e.g., the raters did not have an opportunity to show their reliability so they should get a low score) or whether this is a paradoxical result that invalidates such indexes. The latter camp have advocated alternative metrics, such as S, gamma, or specific agreement. If you decide to go this route, you can calculate them using my agreement package. As below, you can see that all indexes have high observed agreement, but some are also expecting high chance agreement and so adjusted agreement (the index itself such as kappa or alpha) is low.

# install.packages("devtools")
# devtools::install_github("jmgirard/agreement")
> library(agreement)
> codes <- matrix(c(2, 1, 1, 1, 1, 1, 1), nrow = 1)
> cat_adjusted(codes, bootstrap = 0)
#> 
#> Call:
#> cat_adjusted(.data = codes, bootstrap = 0)
#> 
#> Chance-Adjusted Categorical Agreement
#> 
#>                  Observed   Expected   Adjusted
#> identity s          0.714      0.500      0.429
#> identity gamma      0.714      0.245      0.622
#> identity kappa      0.714      0.714      0.000
#> identity pi         0.714      0.755     -0.167
#> identity alpha      0.755      0.755      0.000

I am personally quite fond of category-specific agreement. Below, we can see that the agreement for the 1 category was actually quite good but there was no agreement on the 2 category.

> cat_specific(codes, bootstrap = 0)
#> 
#> Call:
#> cat_specific(.data = codes, bootstrap = 0)
#> 
#> Category-Specific Agreement
#> 
#>     Estimate
#> 1      0.833
#> 2      0.000