I want to calculate the intraclass correlation coefficient (ICC) for an inter-rater assessment with two raters. The design is as follows: Each of the $n$ subjects were assessed by the same 2 raters on several days. Not each subject has the same amount of days where the measurements were done, so the number of replicates (i.e. Days) per person varies.
The dataset is structured as follows: $$ \begin{array}{c|c|c|c} y & \text{Subject ID} & \text{Rater} & \text{Day} \\ \hline \ldots & 1 & 1 & 1 \\ \ldots & 1 & 2 & 1 \\ \ldots & 1 & 1 & 2 \\ \ldots & 1 & 2 & 2 \\ \ldots & 2 & 1 & 1 \\ \ldots & 2 & 2 & 1 \\ \ldots & 2 & 1 & 2 \\ \ldots & 2 & 2 & 2 \\ \ldots & 2 & 1 & 3 \\ \ldots & 2 & 2 & 3 \\ \ldots & \ldots & \ldots & \ldots \\ \end{array} $$
In the above example, subject 1 has measurements on 2 days whereas subject 2 has three days worth of measurements.
For account for the repeated measurements, I want to use a linear mixed model to partition the variance and calculate the ICC. The model I'm considering is the following (using lmer
from the lme4
package):
mod <- lmer(y~1 + (1|Rater) + (1|ID) + (1|ID:Rater) + (1|ID:Day), data = dat)
Random effects:
Groups Name Variance Std.Dev.
ID:Day (Intercept) 2.1437 1.4641
Rater:ID (Intercept) 0.8443 0.9189
ID (Intercept) 0.4949 0.7035
Rater (Intercept) 2.8953 1.7016
Residual 6.9834 2.6426
If I'm not mistaken, the inter-rater agreement can be calculated by:
$$ \mathrm{ICC}_{\mathrm{inter-rater}} = \frac{\sigma_{\mathrm{ID}}^{2} + \sigma_{\mathrm{ID:Day}}^{2}}{\sigma_{\mathrm{ID}}^{2} + \sigma_{\mathrm{ID:Day}}^{2} + \sigma_{\mathrm{Rater}}^{2} + \sigma_{\mathrm{Rater:ID}}^{2} + \sigma_{\mathrm{Residual}}^{2}} = \frac{0.4949 + 2.1437}{0.4949 + 2.1437 + 2.8953 + 0.8443 + 6.9834} = 0.197 $$ I included $\sigma_{\mathrm{ID:Day}}^{2}$ in the numerator because I think this variance component counts as "true" variance of subject-measurements.
I would interpret this number by saying that $19.7\%$ of the variance comes from the variation of the subjects (between and within) and the rest from other sources such as inconsistencies between raters etc.
I have three questions:
- Is the model appropriate for the design of the study?
- Is the formula for the ICC appropriate, or should $\sigma_{\mathrm{ID:Day}}^{2}$ be omitted from the numerator and hence be treated as error-variance?
- How would the model and the formula change if I would consider the 2 raters as fixed in the sense that they are the only two raters I would ever consider (i.e. they weren't selected from an infinite population of possible raters)?