I need help for choosing a proper test for inter-rater reliability. I have three raters (experts) who have analysed the quality of images (10 images from different location). The data is ordinal (image quality is assessed numerically from 1 to 5).
What is the proper test for inter-rater reliability with three raters and ordinal ratings?
For three raters and ordinal data, you can use a chance-adjusted agreement index with ordinal weights. Options include Conger's $\kappa$, Fleiss' $\kappa$, Bennett et al.'s $S$, Krippendorff's $\alpha$, and Gwet's $\gamma$. These can be calculated in MATLAB using the mReliability functions. They can also be calculated in R using the irr package or the AgreeStat functions, sometimes with different names.