Which ICC estimator to use under the fixed effects model?

In the random effects model $$y_{ij} = \mu + \alpha_j + x_{ij} + \epsilon_{ij}$$ the intra-class correlation coefficient is given by $$ICC = \frac{\sigma_{\alpha}}{\sigma_{\alpha}+\sigma_{\epsilon}}$$ and can be estimated by plugging in the variance estimates of the random effects model.

When the number of clusters / groups is small, let's say below 20, we typically do not want to use a random effects model and model $$\alpha_j$$ as fixed effects instead.

What is an accepted estimator for the ICC under the fixed efects model? I am searching the literature but cannot seem to find it.

• ICC is either one-way random model (i.e. only random classes, or subjects, are present, and raters are not indivudually traced) or two-way model (either mixed - with fixed raters, or random - with a random sample of raters). Aug 29, 2020 at 15:58

When the number of clusters / groups is small, let's say below 20, we typically do not want to use a random effects model and model $$\alpha_j$$ as fixed effects instead.

I think most people would disagree that 20 is too few clusters to use a random effects model. While there is no hard and fast rule, 6 seems to be a reasonable consensus.

What is an accepted estimator for the ICC under the fixed efects model? I am searching the literature but cannot seem to find it.

On way to do this is to estimate a model without the grouping factor (call it m0) and then another model with the grouping factor as a fixed effect (call it m1). Then compute the difference in the residual variance and divide it by the residual variance of m0. The idea behind this is that the fixed effects for the grouping factor absorb a certain amount of variance in the response. When they are normally distributed (as is the assumption in linear mixed models) this should be the same amount of variance as that estimated by the random intercepts.

Here is how it can be done in R:

We simulate clustered data with an expected ICC of 0.8 (variance of random intercepts of 4, and residual variance of 1):

> set.seed(2)
> dt <- expand.grid(hospID = 1:10, patientID = 1:20)
> dt$$Y <- 1 > X <- model.matrix(~ 1, data = dt) > myFormula <- "Y ~ 1 + (1 | hospID)" > foo <- lFormula(eval(myFormula), dt) > Z <- t(as.matrix(foo$$reTrms$Zt)) # design matrix for random effects > betas <- 10 # fixed effects (intercept only in this case) > b <- rnorm(10, 0, 2) # random effects (standard deviation of 2, variance of 4) > dt$Y <- X %*% betas + Z %*% b + rnorm(nrow(dt))


Now we fit the linear mixed model:

> library(lme4)
> (lm0 <- lmer(eval(myFormula), dt)) %>% summary()
Random effects:
Groups   Name        Variance Std.Dev.
hospID   (Intercept) 4.011    2.003
Residual             1.188    1.090
Number of obs: 200, groups:  hospID, 10


And we see the estimated variance components are as expected.

Now we fit the models m0 and m1 as described above and compute the ICC from the mixed model and also from the linear models:

> m0 <- lm(Y ~ 1, dt)
> m1 <- lm(Y ~ 1 + as.factor(hospID), dt)

> dt.vc <- as.data.frame(VarCorr(lm0))  # extract the variance components
> (ICC.lmm <- dt.vc[1, 4] / (dt.vc[1, 4] + dt.vc[2, 4]))
[1] 0.7715357
> (ICC.lm <- (var(residuals(m0)) - var(residuals(m1))) / var(residuals(m0)) )
[1] 0.7645219


and these seem to agree well. You can change the seed, change the simulated variances add other fixed effect etc as you see fit.

• Thanks +1. Could you add the R libraries you used to make it reproducible? I am wondering why what you say must hold. What is the math behind it? Also I was just looking into partial Eta-squared which is defined as the sum of squares between groups divided by the sum of the sum of squares between groups and residual sum of squares. This seems to be very similar to the ICC and numerically gives similar results too. Final note: does your approach also work for models with covariates (as in the question). Aug 28, 2020 at 9:37
• Regarding cluster size: this thread provides some alternative perspective stats.stackexchange.com/questions/81606/… Quoted in there, this paper argues that level 2 standard errors are too liberal when groups are 30 or even 50 and lower although regression coefficients are unbiased dspace.library.uu.nl/bitstream/handle/1874/23635/… Aug 28, 2020 at 9:47
• I have edited the answer to add the library and some further explanation of the method. Yes, it works for covariates (other fixed effects) too. Aug 28, 2020 at 9:48
• I've spent most of the last 10 years working with mixed models and they are very robust to small numbers of clusters (~6) in most situations. Aug 28, 2020 at 9:49
• lFormula is also in lme4, it's one of the modular functions for fitting mixed models (just parses the formula and creates the design matrices) Aug 28, 2020 at 9:59

What is an accepted estimator for the ICC under the fixed efects model? I am searching the literature but cannot seem to find it.

That's probably because there's not much to say about the ICC in a standard fixed effects model -- it is necessarily 0 due to the assumptions that the group/cluster effects are "fixed" and thus not random variables (so their variance is 0) and that the errors are uncorrelated.

Mathematical details

Here is some brief background on where the ICC comes from and what it means, with some text repurposed from my answer HERE.

The fixed effects model, as you've written it, is $$y_{ij} = \mu + \alpha_j + x_{ij} + \epsilon_{ij},$$ where the intercepts $$\alpha_j$$ are defined/assumed to be fixed and thus have variance = 0, and the residuals $$\epsilon_{ij}$$ have variance $$\sigma^2_\epsilon$$ (in your question you omitted the square on this term, but here I've added it to be more in line with the usual notation).

Now, the correlation between two random variables $$x$$ and $$y$$ is defined as $$\text{corr} = \frac{\text{cov}(x, y)}{\sqrt{\text{var}(x)\text{var}(y)}}.$$

So to find the intra-class correlation, we use the correlation formula and let our two random variables be two observations (say $$i = 1, 2$$) drawn from the same $$j$$ group,

\begin{aligned} ICC &= \frac{\text{cov}(\mu + \alpha_j + x_{1j} + \epsilon_{1j}, \mu + \alpha_j + x_{2j} + \epsilon_{2j})}{\sqrt{\text{var}(\mu + \alpha_j + x_{1j} + \epsilon_{1j}) \text{var}(\mu + \alpha_j + x_{2j} + \epsilon_{2j})}} \\ &= \frac{\text{cov}(\epsilon_{1j}, \epsilon_{2j})}{\sqrt{\text{var}(\epsilon_{1j}) \text{var}(\epsilon_{2j})}} \\ &= \frac{0}{\sigma^2_\epsilon} \\ &= 0, \end{aligned}

where the numerator simplifies to 0 due to the assumptions that the group/cluster effects are "fixed" and thus not random variables (used on line 2 above) and that the errors are uncorrelated (used on line 3 above).

• I'm thinking about it. Are you sure that the ICC is defined as correlation between $i=1,2$ conditional on $j$? In this wiki entry on the ICC it is stated that the ICC has been discussed in the context of ANOVA (=fixed effects model), so I am not yet certain that there is no useful definition: en.wikipedia.org/wiki/… Sep 2, 2020 at 12:29
• @tomka Yes, I'm sure: the correlation between observations from different parts of the design (in this case, different $j$ clusters) is literally the definition of ICC. Also, your belief that ANOVA = fixed effects model is mistaken. In fact "random effects" (as discussed here) were first developed in an ANOVA context! This classic 1956 paper contains some historical references: projecteuclid.org/download/pdf_1/euclid.aoms/1177728067. There is random effects, fixed effects, and mixed model ANOVA -- see, for example, en.wikipedia.org/wiki/Analysis_of_variance#Classes_of_models Sep 2, 2020 at 13:13