6
$\begingroup$

I understand there are various formulae for calculating an intraclass correlation, and various formulae for calculating a concordance correlation. Is there something that all intraclass correlations have in common, and all concordance correlations have in common?

Is it the case that intraclass correlations are appropriate for a particular type of research question, while concordance correlations are appropriate for a different type of research question? If so, why?

My curiosity in this matter flows out of this question, in which the calculation of a concordance correlation was suggested by the comments, and the calculation of an intraclass correlation was suggested in an answer. However, I see it as a separate question.

$\endgroup$

1 Answer 1

7
$\begingroup$

The modern definition of intraclass correlation (ICC) is a biased estimate of the fraction of the total variance that is due to variation between groups as pertains to the framework of analysis of variance (ANOVA), and random effects models.

What would we use this for? An intraclass correlation (ICC) can be a useful estimate of inter-rater reliability on quantitative data because it is highly flexible. A Pearson correlation can be a valid estimator of interrater reliability, but only when you have meaningful pairings between two and only two raters. What if you have more? What if your raters differ by ratee? This is where ICC comes in (note that if you have qualitative data, e.g. categorical data or ranks, you would not use ICC).

Lin's concordance correlation coefficient (CCC) measures agreement between two variables as a departure from perfect linearity of the $y=x$ type.

$\rho_c = 1 - \frac{{\rm Expected\ orthogonal\ squared\ distance\ from\ the\ diagonal\ }x=y} {{\rm Expected\ orthogonal\ squared\ distance\ from\ the\ diagonal\ }x=y{\rm \ assuming\ independence}}$

CCC is also an inter-rater measurement called an "agreement concordance" rather than an inter-rater "reliability". However, numerically, ICC and CCC can be quite close, sometimes differing in the third decimal place.

One notable difference between ICC and CCC is that CCC can also be used in ordinal (whole number) or nominal scales (named categories), and ICC cannot. However, ICC can be used for more than two raters, and CCC cannot.

$\endgroup$
4
  • $\begingroup$ Okay so if you DO have "qualitative data, e.g. categorical data or ranks", where, to your point "you would not use ICC", what would you use? $\endgroup$ Aug 14, 2021 at 23:50
  • $\begingroup$ @jackisquizzical As above, one can use CCC for that except when there are more than two raters. The cases for more than two raters using CCC and for ICC use for categorical data are not characterized in the literature at the current time. It is possible that CCC could be modified to cover multiple raters, but at present how to do that appears to be unknown. $\endgroup$
    – Carl
    Aug 15, 2021 at 14:55
  • $\begingroup$ @jackisquizzical Theoretically, one could extend CCC by using the L$_2$ 3D norm for the CCC three rater usage, and possibly the L$_1$ norm for 4D (four raters) or more, but so far I haven't seen that done. Note that for higher dimensions, the L$_1$ norm can outperform the L$_2$ norm as a distance measure. But someone would have to explore CCC for more than 2 raters and publish a paper that characterizes the properties of extended CCC before anyone would use it. $\endgroup$
    – Carl
    Aug 15, 2021 at 15:09
  • $\begingroup$ Thanks! Part of the problem with multiple raters is that if the raters disagree (high dispersion or variance), but are centered on the target score, that should be considered. I just posted a related question, and would appreciate your feedback on it. $\endgroup$ Aug 16, 2021 at 13:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.