Significant ICC value, but design effect <2: Is a mixed model approach justified?

Question

An epidemiology study used mixed-level modelling to investigate the effect of individual and household-level exposures on malaria infection (Bannister-Tyrrell M. et al. Importance of household-level risk factors in explaining micro-epidemiology of asymptomatic malaria infections in Ratanakiri Province, Cambodia. Sci Rep. 2018; 8(1). The authors seemed to justify their use of an MLM approach based on the ICC of the null model being statistically significant (ICC: 0.19, p = 0.046).

However, I am reading another paper - A practical guide to multilevel modelling. Peugh J. Journal of School Psychology 48 (2010) 85 - 112 - where the author states that "a non-zero ICC estimate does not necessarily indicate the need for multilevel analyses".

Rather, it is, according to these authors, dependent on the design effect:

DE = 1 + (n - 1)ICC

Where n is the average number of individuals in each cluster, which, in the paper above is 4.2.

Using the numbers from the malaria paper gives me a DE = 1.608.

Peugh goes on to say that "some researchers believe that design effect estimates greater than 2.0 indicate a need for MLM", which, given the values from the malaria paper indicates that an MLM approach was not needed after all.

I have also read another paper (cited in another post on this website): Nezlek J. "An introduction to multilevel modeling for social and personality psychology" Social and Personality Psychology Compass 2/2 (2008): 842–860, where the authors seem to suggest that if the hierarchal structure of the data indicates clustering, an MLM is justified regardless of the ICC (presumably the design effect then also).

I am confused as I do not have the statistical knowledge/background to make a judgement call myself.

The reason I am nitpicking over this is because, like the malaria study above, I have hierarchal data where the ICC is statistically significant but the design effect is only 1.68. And, if I am justified in using traditional single-level logistic regression then my sample size can be smaller, which would help as I am planning a secondary analysis of existing data where the sample size is fairly limited (using an MLM approach, the sample size only has sufficient power to detect a significant difference if the effect size is around 0.7, for a medium effect size (0.5) the power of the study, given an MLM approach, is only 50%).

So, should I be considering the ICC, the DE, or both in my decision-making?

Answer 1

I would favor postulating a model that reflects the design/nature of your study. If you intrinsically have grouped data, it is a good idea to account for the correlations in each group and therefore include the random effect. I would consider leaving it out if you have numerical problems because of near-zero variance for this random effect.

If you really want to test for the random effect, I would use a likelihood ratio test between the model without the random effect and the model that include the random effect. This would correspond to testing whether the random effect's variance is zero. A technical issue with this test is that the null hypothesis is on the boundary of the parameter space. In this case, it is more appropriate to use a mixture of $\chi^2$ distributions to derive the p-value. In practice, this makes a difference if the p-value is close to the significance level.