and of course, time is nested within patient
You could just as easily consider patients nested in occasions, if your research question were about differences between occasions. From the perspective of generalizability theory (GT), your repeated measures are really cross-classified. Your "G-study" design is fully crossed: subjects $\times$ occasions $\times$ raters. This design was discussed in depth by:
- Vangeneugden, T., Laenen, A., Geys, H., Renard, D., & Molenberghs, G. (2005). Applying concepts of generalizability theory on clinical trial data to investigate sources of variation and their impact on reliability. Biometrics, 61(1), 295-304. https://doi.org/10.1111/j.0006-341X.2005.031040.x
In classical test theory (CTT), reliability is merely the ratio of true-score variance to total variance (true + error), where "error" could be estimated as unreliable variance across scale items, raters, or occasions. CTT includes any reliable sources of error in the true score because it would be observed reliably, so estimates of IRR would be inflated by confounding some error across occasions as true-score variance.
GT extended CTT by allowing the error term to be decomposed into different sources of error, thus allowing for the possibility of excluding reliable error from the numerator ("universe-score variance", analogous to true-score variance in CTT). The term "generalizability" in GT is analogous to "reliability" in CTT, and it gives a less ambiguous quality to what is meant by it: How generalizable are your observed "insight" scores across different occasions and/or raters?
The Shrout & Fleiss (1979) notation you mentioned is insufficient for you design because they only discussed "two-facet designs" (subjects $\times$ raters), as did McGraw & Wong (1996) who extended their work and improved their ambiguous notation.
In GT, a single "generalizability coefficient" (G-coef) is an ICC that quantifies reliability/generalizability of your observed scores simultaneously across raters and occasions. For example, if for each subject you chose to represent their insight by averaging their repeated measures across all $N_r=3$ raters and $N_o=4$ occasions ($N_r \times N_o = 12$), then the reliability of those composite scores would be:
$$\text{G-coef} = \frac{\sigma^2_s}{\sigma^2_s + \frac{\sigma^2_{sr}}{N_r} + \frac{\sigma^2_{so}}{N_o} + \frac{\sigma^2_{sro}}{N_r \times N_o}}$$
This is a 3-facet generalization of the 2-facet ICC(2,k) proposed by Shrout & Fleiss (1979), more descriptively labeled ICC(C,k) by McGraw & Wong (1996) because it is a measure of relative Consistency, as opposed to absolute Agreement. But this is a 3-way crossed design, so there is not a single "k" dimension. This is the model you want to fit:
$$\text{insight}=\mu + \beta_p + \beta_r + \beta_o + \beta_{pr} + \beta_{po} + \beta_{ro} + \beta_{pro}$$
where $\mu$ is the grand mean; subscripts indicate effects vary across patients, raters, or occasions (time); and the highest-order term ($\beta_{pro}$) is always confounded with any other source of measurement error. The variance components can be estimated using lme4, where the highest-order term will simply be the default residual ($\varepsilon$):
lmer(insight ~ 1 + (1|patient) + (1|rater) + (1|time) +
(1|patient:rater) + (1|patient:time) + (1|rater:time),
data=d)
In fact, the R package gtheory can be used to automatically calculate the G-coef, specified using lme4 syntax.
If you think it is relevant to still calculate reliability across only one dimension of error (IRR or test-retest reliability), you need to add another source of error variance (the dimension you are ignoring) from the denominator also to the numerator. Find these formulas in Vangeneugden et al. (2005, p. 298, Eqs. 9 and 10), who also discuss and present formulas for ICCs of absolute agreement ("dependability coefficients" in GT), in case you find those relevant to how you will use your observed scores. But note that they do not divide any error by $N$, which estimates reliability of individual scores (i.e., relevant if you do not intend to use repeated measures in the future, and so will not in practice calculate an average to represent each patient's insight).
