3
$\begingroup$

An epidemiology study used mixed-level modelling to investigate the effect of individual and household-level exposures on malaria infection (Bannister-Tyrrell M. et al. Importance of household-level risk factors in explaining micro-epidemiology of asymptomatic malaria infections in Ratanakiri Province, Cambodia. Sci Rep. 2018; 8(1). The authors seemed to justify their use of an MLM approach based on the ICC of the null model being statistically significant (ICC: 0.19, p = 0.046).

However, I am reading another paper - A practical guide to multilevel modelling. Peugh J. Journal of School Psychology 48 (2010) 85 - 112 - where the author states that "a non-zero ICC estimate does not necessarily indicate the need for multilevel analyses".

Rather, it is, according to these authors, dependent on the design effect:

DE = 1 + (n - 1)ICC

Where n is the average number of individuals in each cluster, which, in the paper above is 4.2.

Using the numbers from the malaria paper gives me a DE = 1.608.

Peugh goes on to say that "some researchers believe that design effect estimates greater than 2.0 indicate a need for MLM", which, given the values from the malaria paper indicates that an MLM approach was not needed after all.

I have also read another paper (cited in another post on this website): Nezlek J. "An introduction to multilevel modeling for social and personality psychology" Social and Personality Psychology Compass 2/2 (2008): 842–860, where the authors seem to suggest that if the hierarchal structure of the data indicates clustering, an MLM is justified regardless of the ICC (presumably the design effect then also).

I am confused as I do not have the statistical knowledge/background to make a judgement call myself.

The reason I am nitpicking over this is because, like the malaria study above, I have hierarchal data where the ICC is statistically significant but the design effect is only 1.68. And, if I am justified in using traditional single-level logistic regression then my sample size can be smaller, which would help as I am planning a secondary analysis of existing data where the sample size is fairly limited (using an MLM approach, the sample size only has sufficient power to detect a significant difference if the effect size is around 0.7, for a medium effect size (0.5) the power of the study, given an MLM approach, is only 50%).

So, should I be considering the ICC, the DE, or both in my decision-making?

$\endgroup$
2
  • $\begingroup$ This is a well-written and researched question, with references to the papers mentioned. Can you include a citation for the malaria paper as well? $\endgroup$
    – dipetkov
    Feb 8, 2023 at 21:48
  • 1
    $\begingroup$ Thank you @dipetkov - I have added the citation as you rightly suggested. $\endgroup$ Feb 9, 2023 at 4:24

1 Answer 1

1
$\begingroup$

I would favor postulating a model that reflects the design/nature of your study. If you intrinsically have grouped data, it is a good idea to account for the correlations in each group and therefore include the random effect. I would consider leaving it out if you have numerical problems because of near-zero variance for this random effect.

If you really want to test for the random effect, I would use a likelihood ratio test between the model without the random effect and the model that include the random effect. This would correspond to testing whether the random effect's variance is zero. A technical issue with this test is that the null hypothesis is on the boundary of the parameter space. In this case, it is more appropriate to use a mixture of $\chi^2$ distributions to derive the p-value. In practice, this makes a difference if the p-value is close to the significance level.

$\endgroup$
1
  • $\begingroup$ thanks for the clarity and guidance - it indeed makes sense to go with the intrinsic nature of the study. The technical issue discussion is beyond my current understanding, but I will deal with that later. $\endgroup$ Feb 7, 2023 at 23:51

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.