# intraclass correlation (ICC) to assess interrater reliability with repeated measures in R

I need to estimate the reliability of three raters (A,B,C) rating insight in psychotherapy patients every 10 minutes on an "insight" scale. Given that the experiment lasts 40 minutes, for every patient each rater gives 4 scores, and of course, time is nested within patient.

My dataset looks like this:

patient | time | rater | insight
01      | 01   | A     | 35
01      | 02   | A     | 38
01      | 03   | A     | 31
01      | 04   | A     | 25
01      | 01   | B     | 43
01      | 02   | B     | 41
01      | 03   | B     | 41
01      | 04   | B     | 43
01      | 01   | C     | 35
01      | 02   | C     | 41
01      | 03   | C     | 44
01      | 04   | C     | 42
02      | 01   | A     | 14
02      | 02   | A     | 23
02      | 03   | A     | 18
02      | 04   | A     | 17
02      | 01   | B     | 22
02      | 02   | B     | 24
02      | 03   | B     | 17
02      | 04   | B     | 17
02      | 01   | C     | 35
02      | 02   | C     | 24
02      | 03   | C     | 18
02      | 04   | C     | 12
...


As of my understanding of the matter and of previous answers, I could estimate the agreement between my raters as a "two-way random effects" ICC by fitting the following mixed model:

library(lme4)
fit = lmer(insight ~ (1|rater) + (1|patient/time), data=d)

Random effects:
Groups      Name        Variance Std.Dev.
time:patient (Intercept)  7.330   2.707
patient      (Intercept) 25.418   5.042
coder        (Intercept)  1.860   1.364
Residual                  6.368   2.524
Number of obs: 215, groups:  time:patient, 72; patient, 16; coder, 3


And calculating the ratio between the variance of patients + time (25.42 + 7.33 = 32.75) and the total variance (7.33 + 25.42 + 1.86 + 6.37 = 40.98)

ICC = 32.75/40.98 = 0.799


Still, none of the documentation/answers treated nested levels, and I'm not really savvy with the mathematical notations of statistical papers/manuals, so this solution is a best-guess interpretation.

Questions:

1. Is this procedure correct?
2. Would that be a ICC(2,1) or ICC(2,k) in Shrout and Fleiss (1979) notation?
• Dear Johann. Have you made any progress regarding this question?
– ava
Commented Jul 13, 2021 at 8:58
• @ava no, I wasn't able to find and/or understand more about it. Do you have any idea? Commented Jul 14, 2021 at 11:55

and of course, time is nested within patient

You could just as easily consider patients nested in occasions, if your research question were about differences between occasions. From the perspective of generalizability theory (GT), your repeated measures are really cross-classified. Your "G-study" design is fully crossed: subjects $$\times$$ occasions $$\times$$ raters. This design was discussed in depth by:

• Vangeneugden, T., Laenen, A., Geys, H., Renard, D., & Molenberghs, G. (2005). Applying concepts of generalizability theory on clinical trial data to investigate sources of variation and their impact on reliability. Biometrics, 61(1), 295-304. https://doi.org/10.1111/j.0006-341X.2005.031040.x

In classical test theory (CTT), reliability is merely the ratio of true-score variance to total variance (true + error), where "error" could be estimated as unreliable variance across scale items, raters, or occasions. CTT includes any reliable sources of error in the true score because it would be observed reliably, so estimates of IRR would be inflated by confounding some error across occasions as true-score variance.

GT extended CTT by allowing the error term to be decomposed into different sources of error, thus allowing for the possibility of excluding reliable error from the numerator ("universe-score variance", analogous to true-score variance in CTT). The term "generalizability" in GT is analogous to "reliability" in CTT, and it gives a less ambiguous quality to what is meant by it: How generalizable are your observed "insight" scores across different occasions and/or raters?

The Shrout & Fleiss (1979) notation you mentioned is insufficient for you design because they only discussed "two-facet designs" (subjects $$\times$$ raters), as did McGraw & Wong (1996) who extended their work and improved their ambiguous notation.

In GT, a single "generalizability coefficient" (G-coef) is an ICC that quantifies reliability/generalizability of your observed scores simultaneously across raters and occasions. For example, if for each subject you chose to represent their insight by averaging their repeated measures across all $$N_r=3$$ raters and $$N_o=4$$ occasions ($$N_r \times N_o = 12$$), then the reliability of those composite scores would be:

$$\text{G-coef} = \frac{\sigma^2_s}{\sigma^2_s + \frac{\sigma^2_{sr}}{N_r} + \frac{\sigma^2_{so}}{N_o} + \frac{\sigma^2_{sro}}{N_r \times N_o}}$$

This is a 3-facet generalization of the 2-facet ICC(2,k) proposed by Shrout & Fleiss (1979), more descriptively labeled ICC(C,k) by McGraw & Wong (1996) because it is a measure of relative Consistency, as opposed to absolute Agreement. But this is a 3-way crossed design, so there is not a single "k" dimension. This is the model you want to fit:

$$\text{insight}=\mu + \beta_p + \beta_r + \beta_o + \beta_{pr} + \beta_{po} + \beta_{ro} + \beta_{pro}$$

where $$\mu$$ is the grand mean; subscripts indicate effects vary across patients, raters, or occasions (time); and the highest-order term ($$\beta_{pro}$$) is always confounded with any other source of measurement error. The variance components can be estimated using lme4, where the highest-order term will simply be the default residual ($$\varepsilon$$):

lmer(insight ~ 1 + (1|patient) + (1|rater) + (1|time) +
(1|patient:rater) + (1|patient:time) + (1|rater:time),
data=d)


In fact, the R package gtheory can be used to automatically calculate the G-coef, specified using lme4 syntax.

If you think it is relevant to still calculate reliability across only one dimension of error (IRR or test-retest reliability), you need to add another source of error variance (the dimension you are ignoring) from the denominator also to the numerator. Find these formulas in Vangeneugden et al. (2005, p. 298, Eqs. 9 and 10), who also discuss and present formulas for ICCs of absolute agreement ("dependability coefficients" in GT), in case you find those relevant to how you will use your observed scores. But note that they do not divide any error by $$N$$, which estimates reliability of individual scores (i.e., relevant if you do not intend to use repeated measures in the future, and so will not in practice calculate an average to represent each patient's insight).