Inter-rater agreement with only one subject?

Question

I'm trying to describe inter-rater reliability for a group of 5 researchers working on the same project. We're administering the Hamilton Anxiety rating scale which has 14 items, each item having 4 ordinal levels. Each rater has watched the same video of the same subject being tested and rated it independently. There is only one subject. So we have 5 columns with 14 rows of ordinal data. There is another row that has total HAMA scores for each rater.

To my mind this could be interpreted by simply looking at the data and seeing how much agreement there is. However, some kind of inter-rater statistic has been requested. Does anyone have any recommendations for this? Is something like the Kendall coefficient of concordance or Krippendorff's alpha work here with only one subject?

EDIT: I've noticed that when running something like % agreement in the 'irr' package, it will produce results for '5 raters and 14 subjects'. But of course I have one subject and 14 items.

Cheers for the help

Answer 1

Keeping in mind that the reliability of the results is heavily dependent on the choice of that one subject, the simplest approach is to compute the mean absolute discrepancy between all possible pairs of raters. For your special case this is Gini's mean difference which can be quickly computed in R using the Hmisc package GiniMd function. There is a chapter showing how to generalize this approach to multiple subjects, multiple raters, and multiple assessments within raters - see Chapter 16 of BBR.

Answer 2

You could use any weighted agreement coefficient (e.g., kappa, pi, gamma, alpha, S, or unadjusted) to quantify the amount of item-level agreement between raters. Since you are using R, you could try my work-in-progress agreement package on GitHub. Alternatively, you could use an intraclass correlation coefficient (ICC) to decompose the observed score variance into item, rater, and error terms. However, because you have only one subject, you cannot generalize beyond this subject. So even if agreement or ICC is high, it won't provide evidence that your researchers are reliable for other subjects - only that they are reliable across items.