4
$\begingroup$

I'm analysing data from a paper in which 4 raters; medical professionals, estimate the volume of an organ, they do this twice, for every organ. I want to compute inter-rater and intra-agreement on the results; there are six equations to compute ICC (as listed in Wikipedia):

 Shrout and Fleiss    Name in SPSS
 convention     
 ICC(1,1)             One-way random single measures
 ICC(1,k)             One-way random average measures
 ICC(2,1)             Two-way random single measures (Consistency/Absolute agreement)
 ICC(2,k)             Two-way random average measures (Consistency/Absolute agreement)
 ICC(3,1)             Two-way mixed single measures (Consistency/Absolute agreement)
 ICC(3,k)             Two-way mixed average measures (Consistency/Absolute agreement)

I don't know whether I want ICC(2,1) or ICC(3,1) - the question seems to be 'are these the only raters of interest' or 'were they chosen randomly'? This detail isn't in the paper; another set of medical professionals with relevant expertise could also have done the rating, the most likely scenario is that these 4 were the ones who agreed first to help in the study.

$\endgroup$
4
  • $\begingroup$ This question seems perfectly clear to me. I'm voting to leave open. $\endgroup$ Jan 28 '17 at 2:20
  • $\begingroup$ We really have to let this Shrout-Fleiss notation for ICCs die a quiet death... $\endgroup$ Feb 8 '17 at 23:02
  • $\begingroup$ @JakeWestfall I agree. Take it up with IBM/SPSS. $\endgroup$ Feb 9 '17 at 1:01
  • $\begingroup$ What don't you like about the notation? It seems clear enough to me. $\endgroup$ Jul 31 '17 at 2:32
4
$\begingroup$

From McGraw & Wong (1996), p. 37:

The importance of the random$-$fixed effects distinction is in its effect on the interpretation, but not the calculation, of an ICC. Namely, when levels of the column factor are randomly sampled, one can generalize beyond one's data, but not when they are fixed. In either case, however, the value of the ICC is the same, though one should keep in mind that the population ICCs are defined differently in the two cases.

So you're in luck that the the numerical value of the ICC estimate you calculate will be the same whether you assume ICC(2,1) or ICC(3,1). In terms of which one you should assume in this case, it's hard for an outsider to say. It's common in medical and psychological science for sampling to be less than perfectly random, so ICC(3,1) would be safest. But not being able to generalize beyond the four doctors in that sample may be really problematic depending on what your goal is... So maybe hedge your bets and interpret it as ICC(2,1) with a disclaimer that generalizability to other raters may not be perfect since you are unsure of how the sampling was done to select the four raters.

An alternative (though somewhat sneaky) approach would be to sidestep the issue by adopting the McGraw & Wong (1996) naming convention rather than the one you listed above, which I believe comes from Shrout & Fleiss (1979). In this alternative naming convention, both ICC(2,1) and ICC(3,1) are called ICC(A,1) if the absolute agreement formulation is used or ICC(C,1) if the consistency formulation is used. Of course, the concern about generalizability is still there, and you should still discuss the concern in your paper, but it would prevent you from having to make a definitive choice.

References

McGraw, K. O., & Wong, S. P. (1996). Forming inferences about some intraclass correlation coefficients. Psychological Methods, 1(1), 30–46.

Shrout, P. E., & Fleiss, J. L. (1979). Intraclass correlations: Uses in assessing rater reliability. Psychological Bulletin, 86(2), 420–428.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.