Intraclass correlation with dependent sample units for inter-rater realibility

Question

I have to calculate intraclass correlation to quantify intra and inter-reliability. However, I am not totally sure whether the classical two-way ICC is appropriate because of the correlation among sample units.

In the experiment, each patient has more than one vessel that will be evaluated by 2 readers. The sample unit is the vessel.

The classical two-way ICC is calculated based on the model: $$ y_{ij} = \mu + s_i + r_j + \epsilon_{ij} $$ where $$ s_i \stackrel{i.i.d.}{\sim} N(0, \sigma^2_s) \mbox{ is a random effect for the vessel i;}\\ r_j \stackrel{i.i.d.}{\sim} N(0, \sigma^2_r) \mbox{ is a random effect for the rater j;} \\ \epsilon_{ij} \stackrel{i.i.d.}{\sim} N(0, \sigma^2) \mbox{ is a random effect for the measurement ij,} $$ then $$ ICC = \frac{\sigma^2_s}{\sigma^2_s + \sigma^2_r + \sigma^2}. $$

However, the random effects $s_i$ are not independent since they are from the same patient. I think it is necessary to include a random effect for the patients. In this way, the random effects $s_i$ could be considered independent as described below:

$$ y_{kij} = \mu + p_k + s_i + r_j + \epsilon_{kij} $$ where $$ p_k \stackrel{i.i.d.}{\sim} N(0, \sigma^2_p) \mbox{ is a random effect for the patient k;}\\ s_i \stackrel{i.i.d.}{\sim} N(0, \sigma^2_s) \mbox{ is a random effect for the vessel i;}\\ r_j \stackrel{i.i.d.}{\sim} N(0, \sigma^2_r) \mbox{ is a random effect for the rater j;} \\ \epsilon_{kij} \stackrel{i.i.d.}{\sim} N(0, \sigma^2) \mbox{ is a random effect for the measurement kij,} $$ then $$ ICC = \frac{\sigma^2_s}{\sigma^2_p + \sigma^2_s + \sigma^2_r + \sigma^2}. $$

Does it make sense?

Answer 1

I think I found my answer going back to the ICC definition:

The ICC is defined as the correlation between two observations. The correlation between two images from the same patient is given by

$$ \begin{eqnarray} ICC &=& Corr(y_{kij}, y_{kij'}) \\ &=& \displaystyle\frac{Cov(\mu + p_k + s_i + r_j + \epsilon_{kij}, \mu + p_k + s_i + r_{j'} + \epsilon_{kij})}{\sqrt{Var(y_{kij})Var(y_{kij'})}} \\ &=& \displaystyle\frac{Cov(p_k, p_k) + Cov(s_i, s_i)}{Var(y_{kij})} \\ &=& \displaystyle\frac{\sigma^2_p + \sigma^2_s}{\sigma^2_p + \sigma^2_s + \sigma^2_r + \sigma^2}, \end{eqnarray} $$

which is the same as the classical two-way ICC. The only difference is that I would be partitioning $\sigma^2_s$ in two parts given by $\sigma^2_s$ and $\sigma^2_p$.

However, the 95% confidence interval would be different because requires the independence assumption of the observations. Bootstrap seems a solution. The package lme4 has a function bootMer() that samples the random effects which can be considered independent.