I've asked about the mechanics of Inter-rater reliability before, but I guess I should have asked this question instead: If I have a bunch of raters and individuals (who are being judged on performance), but they come in blocks, so Group 1 has 5 individuals and 3 raters. Group 2 has 10 individuals and 4 raters, and none of these raters overlap, what is the best statistical method to compare these groups?
I probably won't get to compare the individuals to one another if they're in different blocks, but the idea is, I want to make sure that no one block of raters was more or less in agreement than the in the other blocks of raters. I want to identify those blocks of raters where disagreement is really high about the individuals--and potentially identifying raters that are overly strict, bringing an individual's average score down.
I have continuous data, so the Scores for individuals range from 1.000 to 5.000, averaging 2.5555 or something like that.
Is there a better way to compare and analyze the raters than IRR? Maybe graphically? I use Stata.