Intraclass correlation coefficient (ICC) for two raters using a mixed effects model for a design with repeated measurements

Question

I want to calculate the intraclass correlation coefficient (ICC) for an inter-rater assessment with two raters. The design is as follows: Each of the $n$ subjects were assessed by the same 2 raters on several days. Not each subject has the same amount of days where the measurements were done, so the number of replicates (i.e. Days) per person varies.

The dataset is structured as follows: $$ \begin{array}{c|c|c|c} y & \text{Subject ID} & \text{Rater} & \text{Day} \\ \hline \ldots & 1 & 1 & 1 \\ \ldots & 1 & 2 & 1 \\ \ldots & 1 & 1 & 2 \\ \ldots & 1 & 2 & 2 \\ \ldots & 2 & 1 & 1 \\ \ldots & 2 & 2 & 1 \\ \ldots & 2 & 1 & 2 \\ \ldots & 2 & 2 & 2 \\ \ldots & 2 & 1 & 3 \\ \ldots & 2 & 2 & 3 \\ \ldots & \ldots & \ldots & \ldots \\ \end{array} $$

In the above example, subject 1 has measurements on 2 days whereas subject 2 has three days worth of measurements.

For account for the repeated measurements, I want to use a linear mixed model to partition the variance and calculate the ICC. The model I'm considering is the following (using lmer from the lme4 package):

mod <- lmer(y~1 + (1|Rater) + (1|ID) + (1|ID:Rater) + (1|ID:Day), data = dat)

Random effects:
 Groups        Name        Variance Std.Dev.
 ID:Day        (Intercept) 2.1437   1.4641  
 Rater:ID      (Intercept) 0.8443   0.9189  
 ID            (Intercept) 0.4949   0.7035  
 Rater         (Intercept) 2.8953   1.7016  
 Residual                  6.9834   2.6426  

If I'm not mistaken, the inter-rater agreement can be calculated by:

$$ \mathrm{ICC}_{\mathrm{inter-rater}} = \frac{\sigma_{\mathrm{ID}}^{2} + \sigma_{\mathrm{ID:Day}}^{2}}{\sigma_{\mathrm{ID}}^{2} + \sigma_{\mathrm{ID:Day}}^{2} + \sigma_{\mathrm{Rater}}^{2} + \sigma_{\mathrm{Rater:ID}}^{2} + \sigma_{\mathrm{Residual}}^{2}} = \frac{0.4949 + 2.1437}{0.4949 + 2.1437 + 2.8953 + 0.8443 + 6.9834} = 0.197 $$ I included $\sigma_{\mathrm{ID:Day}}^{2}$ in the numerator because I think this variance component counts as "true" variance of subject-measurements.

I would interpret this number by saying that $19.7\%$ of the variance comes from the variation of the subjects (between and within) and the rest from other sources such as inconsistencies between raters etc.

---

I have three questions:

1. Is the model appropriate for the design of the study?
2. Is the formula for the ICC appropriate, or should $\sigma_{\mathrm{ID:Day}}^{2}$ be omitted from the numerator and hence be treated as error-variance?
3. How would the model and the formula change if I would consider the 2 raters as fixed in the sense that they are the only two raters I would ever consider (i.e. they weren't selected from an infinite population of possible raters)?

Answer 1

1. Is the model appropriate for the design of the study?

I think not. The issue is that you have only 2 raters. You are asking the software to estimate the variance of a normally distributed variable using only 2 observations, so any estimate of a variance for this variable, and any statistic that uses it, should be highly suspect.

2. Is the formula for the ICC appropriate, or should $\sigma_{\mathrm{ID:Day}}^{2}$ be omitted from the numerator and hence be treated as error-variance?

Yes, I think your formula is appropriate.

3. How would the model and the formula change if I would consider the 2 raters as fixed in the sense that they are the only two raters I would ever consider (i.e. they weren't selected from an infinite population of possible raters)?

In light of my answer to 1. above, I think you should take this approach anyway. Whether they can be considered samples from a large population is only one of the considerations in choosing whether to model a factor as fixed or random.

The formula then becomes:

$$ \mathrm{ICC}_{\mathrm{inter-rater}} = \frac{\sigma_{\mathrm{ID}}^{2} + \sigma_{\mathrm{ID:Day}}^{2}}{\sigma_{\mathrm{ID}}^{2} + \sigma_{\mathrm{ID:Day}}^{2} + \sigma_{\mathrm{Residual}}^{2}} $$