0
$\begingroup$

I'm trying to describe inter-rater reliability for a group of 5 researchers working on the same project. We're administering the Hamilton Anxiety rating scale which has 14 items, each item having 4 ordinal levels. Each rater has watched the same video of the same subject being tested and rated it independently. There is only one subject. So we have 5 columns with 14 rows of ordinal data. There is another row that has total HAMA scores for each rater.

To my mind this could be interpreted by simply looking at the data and seeing how much agreement there is. However, some kind of inter-rater statistic has been requested. Does anyone have any recommendations for this? Is something like the Kendall coefficient of concordance or Krippendorff's alpha work here with only one subject?

EDIT: I've noticed that when running something like % agreement in the 'irr' package, it will produce results for '5 raters and 14 subjects'. But of course I have one subject and 14 items.

Cheers for the help

$\endgroup$
0
$\begingroup$

You could use any weighted agreement coefficient (e.g., kappa, pi, gamma, alpha, S, or unadjusted) to quantify the amount of item-level agreement between raters. Since you are using R, you could try my work-in-progress agreement package on GitHub. Alternatively, you could use an intraclass correlation coefficient (ICC) to decompose the observed score variance into item, rater, and error terms. However, because you have only one subject, you cannot generalize beyond this subject. So even if agreement or ICC is high, it won't provide evidence that your researchers are reliable for other subjects - only that they are reliable across items.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.