Randomization scheme for rating images and assessing inter-rater reliability

Question

Apologies for the basic question, but I'm new to the area and I'm working on a research project on rating images, and assessing the reliability of these ratings, with the following structure:

There are 973 images, each of which has to be rated 3 times. These 2919 total ratings are to be allocated as evenly as possible between 5 raters (e.g., so that a rater is unlikely to rate the same image 3 times). Each rater will rate the "goodness" of their images on a 3-point scale ("Pass", "Maybe", "Fail"). Finally, we also want to assess the inter-rater reliability of the ratings.

If anyone knows of (and how to implement) a good (or, better yet, optimal) randomization scheme to allocate the images to the raters in a way that would facilitate the assessment of the reliability of the ratings, any advice would be greatly appreciated.

Answer 1

If you want to calculate inter-rater reliability, you should prevent each individual rater from assigning multiple ratings to the same image. That is, each image should be rated by three different raters. Aside from that, any randomization scheme would be fine. Maybe try to ensure that all five raters end up rating similar numbers of images (e.g., $2919/5=583.8$).

Also, be sure to use a formulation of agreement that allows for multiple, different raters. A generalized formula for agreement, Fleiss' kappa coefficient, or Bennett et al.'s S score with ordinal weights would work well.