I am trying to calculate the Intraclass Correlation for a rater study using R and the library lme4 and the function lmer. The data has the following design: The same 6 raters (at least 4) are rating 25 horses live and all raters are rating a subset of 10 horses on video.

The two way random model applied:

m1 <- lmer(Score ~ -1 + (1|HorseID) + (1|RaterID) + Time, data=mydata)

The absolute agreement ICC is calculated using the estimated coefficients:

xVars <- function(model) {
exvars = lme4::VarCorr(model)
vars = c(exvars$HorseID[1,1], exvars$RaterID[1,1], attr(exvars,"sc")^2) 
names(vars) <- c('item var', 'judge var', 'residual var')
vars }

# helper function for ICC(k) variations

icck <- function(variances, k=1) {
icc = variances[1] / (variances[1] + (variances[2] + variances[3]) / k)
names(icc) = c(paste('ICC', k, sep=''))
icc }

And the ICC as:

ICC.m1 <- icck(xVars(m1))

I would like to add:

  • Confidence Intervals for the ICC
  • Cronbach's alpha

But I can't figure out at smart way of doing so? Help would be greatly appreciated!!

  • $\begingroup$ You don't need a mixed-effect model to compute Cronbach alpha, only the variance of the total score and that of individual items. However, the average measures ICC for the 2-way mixed model (ICC(2,k)) should be close enough. Re. ICC, you can use the bootstrap or asymptotic formula, e.g., Comparison of confidence interval methods for an intra-class correlation coefficient (ICC). (but there are many other approaches that were suggested in the past, especially for rater reliability). $\endgroup$ – chl Oct 25 '20 at 9:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.