How do I calculate the confidence interval of an ICC?

Question

Here's the output from my the summary of lmer function which I used to calculate the ICC.

   Linear mixed model fit by REML ['lmerMod']
   Formula: CareChange ~ 1 + (1 | PROVIDER)
      Data: MEA_data_1

   REML criterion at convergence: 35.2

   Scaled residuals: 
       Min      1Q  Median      3Q     Max 
   -0.3093 -0.2829 -0.2711 -0.2599  3.6206 

   Random effects:
    Groups            Name        Variance  Std.Dev.
    PROVIDER          (Intercept) 0.0003096 0.01759 
    Residual                      0.0660667 0.25703 
   Number of obs: 239, groups:  PROVIDER_CALENDAR, 14

   Fixed effects:
        Estimate Std. Error t value
        (Intercept)  0.07119    0.01742   4.086

   ICC = 0.0003096 /(0.0003096 +0.0660667 ) = 0.004664315

I did not find any literature that shows how to calculate the CI for this ICC value. Any help on this issue is much appreciated.

Answer 1

A straightforward way to calculate a confidence interval would be to create a bootstrapped distribution, then obtain the relevant quantiles from that distribution. This can be done as a parametric or nonparametric bootstrap, depending on what assumptions you're comfortable with. Generally, if you accept a Normal distribution for the random effects, a parametric bootstrap is also appropriate.

Since you're using R, the lme4 package offers a very nice parametric bootstrap function, bootMer() to calculate a bootstrapped distribution of any statistic of interest derived from a random effects model. The parametric method is fully implemented, which simulates the model of interest using new values of the random effects with each iteration. New values are drawn from a Normal distribution using parameters derived from the mixed model. See the documentation in bootMer() for some detail on control over what effects get bootstrapped. Set your own number of iterations, but 1000+ is a good rule of thumb.

To calculate this in R, you need to fit the random effects model and pass a function to bootMer() that calculates the statistic of interest. An example implementation is below:

Example Code using R

#Make some mocked data
library(lme4)
library(reshape2)
set.seed(2024)  #For the Bell Riots
id <- factor(seq(1, 15))
id.mu <- rnorm(15, 10, 5)
mydat <- NULL
for (a in 1:length(id)){
  score <- rnorm(2, id.mu[a], 3)
  id.fr <- data.frame(id=id[a], score1=score[1], score2=score[2])
  mydat <- rbind(mydat, id.fr)
}
mydat <- melt(mydat, id.vars='id', value.name='score')

#Create function to calculate ICC from fitted model
calc.icc <- function(y) {
  sumy <- summary(y)
  (sumy$varcor$id[1]) / (sumy$varcor$id[1] + sumy$sigma^2)
}

#Fit the random effects model and calculate the ICC
mymod <- lmer(score ~ 1 + (1|id), data=mydat)
summary(mymod)
calc.icc(mymod)

#Calculate the bootstrap distribution
boot.icc <- bootMer(mymod, calc.icc, nsim=1000)

#Draw from the bootstrap distribution the usual 95% upper and lower confidence limits
quantile(boot.icc$t, c(0.025, 0.975))

Answer 2

I had the same need, here's another solution that I ended up with using the irr package to bootstrap this more easily. (And less clever too, but if it can help future users ...)

I know the package gives you a confidence interval as well as the ICC package but if you're bothered because of the distribution and need a bootstrap:

#create the matrix needed for your task
matrix <- select(dataframe, var1, var2) 

#you need to define the model, type and unit according to the question you're asking, here's for the example
irr:icc(matrix, model ="twoway", type = "agreement", unit = "single") 

#New fuction, [7] is to get the value we want (icc in this case)
icc.boot <- function(data,x) {irr::icc(data[x,], model ="twoway", type = "agreement", unit = "single")[[7]]} 

#bootstsrap distribution with 10 000 samples in this example
boot <- boot(matrix,icc.boot, 10000) 

# get the confidence interval 
quantile(boot$t,c(0.025,0.975)) 

# If you're unsure of the number of samples to choose, you can get a look of the distribution of the bootstrap 
hist(boot) 

I hope it helps.