Assessing independence of observations using intraclass correlation when some groups have small group sample sizes Context
I was talking to a researcher in the following situation.


*

*Participants (n = 500) were sampled from schools.

*Participants came from around 50 different schools.

*The number of participants per school varied with some schools supplying 20 or 30 participants, but a few schools only supplying 3 or 4 or 5 participants.

*The researcher was trying to assess whether there was a substantial violation of the independence of observations assumption. Thus, they were looking at the intra-class correlation of core outcome variables based on the effect of school.


Question


*

*Is it problematic to include groups in an intra-class correlation analysis with small numbers per group? If so, what are the implications of this? What might a rule of thumb regarding a minimum number for inclusion be?

*To take it to the extreme, what is some groups only supplied a single participant?

*If there are groups with samples sizes below a given threshold, what is a good subsequent course of action? remove group from analysis? collapse small groups into an "other" group?

*How would any recommendations given relate to the assessment of the independence of observations assumption?

 A: I'm not sure about the metric of the ICC itself (I have never seen anyone report this metric for inferential purposes, only for description), but I do not believe many modelling strategies will be greatly impacted by the instance of many small groups. This is because random effects modelling takes into account the sample size of the groups, by "shrinking" the estimated group variances by their sample sizes. As a note when I refer to fixed or random effects, it is in line with the definitions layed out here.
One way to assess this is to examine the outcome of interest as deviations from the group means as oppossed to the original metric. So if $y_{ij}$ is the variable $y$ for observation $i$ within group $j$, from this variable subtract the mean of group $j$, and graph a scatterplot of those deviations versus the group size. You would expect this scatterplot to show heteroscedasticity (as the mean of smaller group sizes should be less representative of the observations within the group), and have a wider variance for smaller groups. If the opposite occurs, this suggests that the smaller groups are more homogenous, and might be evidence that the independence of observations is violated and is directly related to group size (e.g. those in smaller groups tend to be more similar to each other than those in larger groups).
If anything small groups should inflate the ICC. When there is only 1 observation, all of the variance for that observation is attributed to the group level mean, and if all groups only had 1 observation, the ICC would be 1 (i.e. there is no within group variation, only between groups).
Also as a note, frequently to estimate some relationship it is not that the observations need to be independent, it is simply that the model residuals need to be independent. Hence the whole reason to fit multi-level models.
Stephen Raudenbush has a book chapter, Many Small Groups that may be of interest (I see a PDF of the whole book can be found here). The chapter is mainly about how to estimate models with many small groups and potential problems that can arise. This is really only pertinent though if you want to estimate random effects models. If you are simply interested in fixed effects models it is largely unproblematic.
Also I have found the tutorials developed by the Centre for Multilevel Modelling to be very useful introductions to the subject material (very gentle, especially compared to the Raudenbush chapter I just cited!)
