Apologies for the basic question, but I'm new to the area and I'm working on a research project on rating images, and assessing the reliability of these ratings, with the following structure:
There are 973 images, each of which has to be rated 3 times. These 2919 total ratings are to be allocated as evenly as possible between 5 raters (e.g., so that a rater is unlikely to rate the same image 3 times). Each rater will rate the "goodness" of their images on a 3-point scale ("Pass", "Maybe", "Fail"). Finally, we also want to assess the inter-rater reliability of the ratings.
If anyone knows of (and how to implement) a good (or, better yet, optimal) randomization scheme to allocate the images to the raters in a way that would facilitate the assessment of the reliability of the ratings, any advice would be greatly appreciated.