I am currently working on intraclass correlation in the context of inter-rater reliability. There are different forms of ICC (Shrout and Fleiss, 1979; McGraw and Wong, 1999) and according to the literature three questions need to be answered to make the right choice between these different forms:
- Which model to choose (one-way model, two-way random model, two-way mixed model)?
- Are you interested in consistency or agreement?
- Are you interested in the single or average ICC?
I have a design and question that suggests a two-way random model. In the study, different raters have rated several objects on the same construct. Accordingly, I would set up the population model for the single ICC(2,1) (McGraw and Wong, 1999) as follows.
Model:
$$ y_{ij} = \mu + o_i + r_j + e_{ij}\,,$$
with $o = $ object (random factor) and $r = $ rater (also random factor).
ICC:
$$ ICC(2,1) = \frac{\sigma_0^2}{\sigma_0^2+\sigma_r^2+\sigma_e^2}$$
As far as I know, the ICC can be calculated using both the MS of an ANOVA model and the variance components of a mixed model. The mixed model approach seems simpler to me and also seems to handle missing values better. Therefore, I am following this approach.
My current question relates to the difference between single and average ICC.
Question 1 – Formula of average ICC:
- Calculating the average ICC To calculate the average ICC (e.g. $ICC(2,k)$), the formula is adapted as follows:
$$ ICC(2,k) = \frac{\sigma_0^2}{\sigma_0^2+(\sigma_r^2+\sigma_e^2)/k}$$
$k$ corresponds to the number of raters in this formula. Can anyone explain this formula to me? Why is the variance of the assessors and the residual variance divided by k? Does this represent the expected value of the variance of the average?
Question 2 – Interpretation of ICC
In the above example, is it correct for me to interpret the ICC as follows:
- Simple: the expected correlation of the scores when two randomly selected raters rate the same item.
- Average: The expected correlation of the average values from two random samples of size k.
