Inter-rater reliability or another method?

Question

I've asked about the mechanics of Inter-rater reliability before, but I guess I should have asked this question instead: If I have a bunch of raters and individuals (who are being judged on performance), but they come in blocks, so Group 1 has 5 individuals and 3 raters. Group 2 has 10 individuals and 4 raters, and none of these raters overlap, what is the best statistical method to compare these groups?

I probably won't get to compare the individuals to one another if they're in different blocks, but the idea is, I want to make sure that no one block of raters was more or less in agreement than the in the other blocks of raters. I want to identify those blocks of raters where disagreement is really high about the individuals--and potentially identifying raters that are overly strict, bringing an individual's average score down.

I have continuous data, so the Scores for individuals range from 1.000 to 5.000, averaging 2.5555 or something like that.

Is there a better way to compare and analyze the raters than IRR? Maybe graphically? I use Stata.

Answer 1

I would try with one or all of theese Light's kappa, Fleiss's kappa, Krippendorff's alpha. Hava a look here

If the "items" beeing rated are somehow similiar you can always try to impute missing values (using for example kNN) from one block to another to guess how the judges from block A would rate item from block B and vice versa.