I have a data set of 11,000+ distinct items, each of which was classified on a nominal scale by at least 3 different raters on Amazon's Mechanical Turk.
88 different raters provided judgments for the task, and no one rater completed more about 800 judgments. Most provided significantly fewer than that.
My question is this:
I would like to calculate some measure of inter-rater reliability for the ratings, something better than a simply looking at consensus. I believe, however, that Fleiss Kappa, which is the measure I know best, would require a consistent group of raters for the entire set of items, and so I cannot use Fleiss Kappa to check IRR with my data. Is this correct? Is there another method I could use?
Any advice would be much appreciated!