Inter-rater Reliability (Krippendorf's Alpha) for raters of a single observed unit?

Question

I've got some crowdsourced data rating avatar images for various emotion & personality traits, and based on previous work by a fellow postdoc I'm trying to analyze inter-rater reliability on a per-image basis: I want to show that some images are less agreed-upon than others, so I need to compute per-image inter-rater reliability. Unfortunately, as far as I can tell Krippendorf's alpha isn't usable, because it computes expected differences across all units, so when there's only one unit, the expected differences always exactly cancel the observed differences.

Essentially, I'm in the same boat as this (unanswered) question: How to calculate inter-rater reliability for just one sample?

I plan to fall back on a simple percent agreement calculation, but I wonder if there's a better/standard solution to this problem. I've read through e.g., this nice summary of various IRR statistics, and it looks like Fleiss' Kappa might apply, although one complication is that I've got ordinal data rather than nominal data, although treating it as nominal wouldn't be the end of the world.

TL;DR: would like advice on best inter-rater reliability method when there is only one unit.

(Also: I'd love to be told I'm wrong about the applicability of Krippendorf's alpha, since it's otherwise perfect, and includes specialized treatment for ordinal data).

Update: I tried using Gwet's R functions suggested below and as I expected Krippendorf's alpha doesn't seem to work with a single row (unless I'm doing something wrong, here's a MWE):

> source("agree.coeff3.dist.r")
> split <- data.frame(one=c(1), two=c(0), three=c(1), four=c(3), 
five=c(1))> split
  one two three four five
1   1   0     1    3    1
> fleiss.kappa.dist(split)
Fleiss' Kappa Coefficient
==========================
Percent agreement: 0.2 Percent chance agreement: 0.3333333 
Fleiss kappa coefficient: -0.2 Standard error: NaN 
95 % Confidence Interval: ( NaN , NaN )
P-value:  NaN 
Warning messages:
1: In qt(1 - (1 - conflev)/2, n - 1) : NaNs produced
2: In qt(1 - (1 - conflev)/2, n - 1) : NaNs produced
> krippen.alpha.dist(split)
Error in weights.mat * (pi.vec %*% t(pi.vec)) : non-conformable arrays

It looks like Fleiss' kappa is working though, although there are some issues with the standard error and confidence interval.

Fleiss' kappa thus seems to be a usable alternative.

Answer 1

You can calculate any of the chance-adjusted indexes of reliability (e.g., alpha, kappa, or pi) for a single item; you just need to use the right formula. With almost all of these indexes, you can account for the ordinal relationship between categories (this is not a unique feature of alpha anymore). To do these calculations, you can use my MATLAB functions or Gwet's R functions.

However, just because it is possible to compute single-item reliability indexes doesn't mean this is the best approach. I actually suspect that weighted agreement would be clearer in your case than a chance-adjusted index. I'd also recommend trying to model the variation in ratings for all images at once rather than on an image-by-image basis, but I don't know much about your actual goals here.