I need help for choosing a proper test for inter-rater reliability. I have three raters (experts) who have analysed the quality of images (10 images from different location). The data is ordinal (image quality is assessed numerically from 1 to 5).
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
For three raters and ordinal data, you can use a chance-adjusted agreement index with ordinal weights. Options include Conger's $\kappa$, Fleiss' $\kappa$, Bennett et al.'s $S$, Krippendorff's $\alpha$, and Gwet's $\gamma$. These can be calculated in MATLAB using the mReliability functions. They can also be calculated in R using the irr package or the AgreeStat functions, sometimes with different names.
answered Apr 13, 2016 at 0:31