How to perform inter-rater reliability with multiple raters, different raters per participant, and possible changes over time? Participants were rated twice, with the 2 ratings separated by 3 years. For most participants the ratings were done by different raters, but for some (< 10%) the same rater performed both ratings. There were 8 raters altogether, with 2 doing ratings at both time points. 
Now, since the ratings were of an aspect of ability with a hypothetical "correct" value, then absolute agreement between raters is of interest, rather than consistency. However, since the ratings were taken 3 years apart, there might have been (and probably was) some real change in the ability. 


*

*What would be the best test of reliability in this case?

*I'm leaning towards an intra-class correlation, but is ICC1 the best I can do with these data?
 A: How are you planning to account for the fact that some ratings were done by the same rater? Off the top of my head, I can't think of any measures that take that into account when it isn't consistently done. After all, if you compare the same rater twice, you're looking at consistency; if you compare two raters, you're looking at agreement. So, when you say you want to evaluate the "reliability", it's not totally clear what you are seeking to evaluate.
If you believe that the ability level of the subjects is likely to have changed, it's also important to consider how you can account for that fact. Do you have some gold-standard measurement to compare the raters against?
So, in summary, before you can assess how reliable the raters are, you need to answer two key questions:


*

*How can you quantify and correct for change between the timepoints attributed to legitimate changes in ability, instead of poor consistency in rating?

*Are you principally interested in how often the raters agree with each other, or in how consistently they apply the ratings?

