3
$\begingroup$

I need help for choosing a proper test for inter-rater reliability. I have three raters (experts) who have analysed the quality of images (10 images from different location). The data is ordinal (image quality is assessed numerically from 1 to 5).

$\endgroup$
1
$\begingroup$

For three raters and ordinal data, you can use a chance-adjusted agreement index with ordinal weights. Options include Conger's $\kappa$, Fleiss' $\kappa$, Bennett et al.'s $S$, Krippendorff's $\alpha$, and Gwet's $\gamma$. These can be calculated in MATLAB using the mReliability functions. They can also be calculated in R using the irr package or the AgreeStat functions, sometimes with different names.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.