I've got some crowdsourced data rating avatar images for various emotion & personality traits, and based on previous work by a fellow postdoc I'm trying to analyze inter-rater reliability on a per-image basis: I want to show that some images are less agreed-upon than others, so I need to compute per-image inter-rater reliability. Unfortunately, as far as I can tell Krippendorf's alpha isn't usable, because it computes expected differences across all units, so when there's only one unit, the expected differences always exactly cancel the observed differences.
Essentially, I'm in the same boat as this (unanswered) question: How to calculate inter-rater reliability for just one sample?
I plan to fall back on a simple percent agreement calculation, but I wonder if there's a better/standard solution to this problem. I've read through e.g., this nice summary of various IRR statistics, and it looks like Fleiss' Kappa might apply, although one complication is that I've got ordinal data rather than nominal data, although treating it as nominal wouldn't be the end of the world.
TL;DR: would like advice on best inter-rater reliability method when there is only one unit.
(Also: I'd love to be told I'm wrong about the applicability of Krippendorf's alpha, since it's otherwise perfect, and includes specialized treatment for ordinal data).
Update: I tried using Gwet's R functions suggested below and as I expected Krippendorf's alpha doesn't seem to work with a single row (unless I'm doing something wrong, here's a MWE):
> source("agree.coeff3.dist.r")
> split <- data.frame(one=c(1), two=c(0), three=c(1), four=c(3),
five=c(1))> split
one two three four five
1 1 0 1 3 1
> fleiss.kappa.dist(split)
Fleiss' Kappa Coefficient
==========================
Percent agreement: 0.2 Percent chance agreement: 0.3333333
Fleiss kappa coefficient: -0.2 Standard error: NaN
95 % Confidence Interval: ( NaN , NaN )
P-value: NaN
Warning messages:
1: In qt(1 - (1 - conflev)/2, n - 1) : NaNs produced
2: In qt(1 - (1 - conflev)/2, n - 1) : NaNs produced
> krippen.alpha.dist(split)
Error in weights.mat * (pi.vec %*% t(pi.vec)) : non-conformable arrays
It looks like Fleiss' kappa is working though, although there are some issues with the standard error and confidence interval.
Fleiss' kappa thus seems to be a usable alternative.