Return to Question

An epidemiology study used mixed-level modelling to investigate the effect of individual and household-level exposures on malaria infection. The authors justified their use of an MLM approach based on the ICC of the null model being statistically significant (ICC: 0.19, p = 0.046).

However, I am reading another paper - A practical guide to multilevel modelling. Peugh J. Journal of School Psychology 48 (2010) 85 - 112 - where the author states that "a non-zero ICC estimate does not necessarily indicate the need for multilevel analyses".

Rather, it is, according to these authors, dependent on the design effect:

DE = 1 + (n - 1)ICC

Where n is the average number of individuals in each cluster, which, in the paper above is 4.2.

Using the numbers from the malaria paper gives me a DE = 1.608.

Peugh goes on to say that "some researchers believe that design effect estimates greater than 2.0 indicate a need for MLM", which, given the values from the malaria paper indicates that an MLM approach was not needed after all.

I have also read another paper (cited in another post on this website): Nezlek J. "An introduction to multilevel modeling for social and personality psychology" Social and Personality Psychology Compass 2/2 (2008): 842–860, where the authors seem to suggest that if the hierarchal structure of the data indicates clustering, an MLM is justified regardless of the ICC (presumably the design effect then also).

I am confused as I do not have the statistical knowledge/background to make a judgement call myself.

The reason I am nitpicking over this is because, like the malaria study above, I have hierarchal data where the ICC is statistically significant but the design effect is only 1.68. And, if I am justified in using traditional single-level logistic regression then my sample size can be smaller, which would help as I am planning a secondary analysis of existing data where the sample size is fairly limited (using an MLM approach, the sample size only has sufficient power to detect a significant difference if the effect size is around 0.7, for a medium effect size (0.5) the power of the study, given an MLM approach, is only 50%).

So, should I be considering the ICC, the DE, or both in my decision-making?

An epidemiology study used mixed-level modelling to investigate the effect of individual and household-level exposures on malaria infection (Bannister-Tyrrell M. et al. Importance of household-level risk factors in explaining micro-epidemiology of asymptomatic malaria infections in Ratanakiri Province, Cambodia. Sci Rep. 2018; 8(1). The authors seemed to justify their use of an MLM approach based on the ICC of the null model being statistically significant (ICC: 0.19, p = 0.046).

However, I am reading another paper - A practical guide to multilevel modelling. Peugh J. Journal of School Psychology 48 (2010) 85 - 112 - where the author states that "a non-zero ICC estimate does not necessarily indicate the need for multilevel analyses".

Rather, it is, according to these authors, dependent on the design effect:

DE = 1 + (n - 1)ICC

Where n is the average number of individuals in each cluster, which, in the paper above is 4.2.

Using the numbers from the malaria paper gives me a DE = 1.608.

Peugh goes on to say that "some researchers believe that design effect estimates greater than 2.0 indicate a need for MLM", which, given the values from the malaria paper indicates that an MLM approach was not needed after all.

I have also read another paper (cited in another post on this website): Nezlek J. "An introduction to multilevel modeling for social and personality psychology" Social and Personality Psychology Compass 2/2 (2008): 842–860, where the authors seem to suggest that if the hierarchal structure of the data indicates clustering, an MLM is justified regardless of the ICC (presumably the design effect then also).

I am confused as I do not have the statistical knowledge/background to make a judgement call myself.

The reason I am nitpicking over this is because, like the malaria study above, I have hierarchal data where the ICC is statistically significant but the design effect is only 1.68. And, if I am justified in using traditional single-level logistic regression then my sample size can be smaller, which would help as I am planning a secondary analysis of existing data where the sample size is fairly limited (using an MLM approach, the sample size only has sufficient power to detect a significant difference if the effect size is around 0.7, for a medium effect size (0.5) the power of the study, given an MLM approach, is only 50%).

So, should I be considering the ICC, the DE, or both in my decision-making?

I am reading a paper that is using mixed-level modelling to investigate the effect of individual (level 1) and household-level (level 2) exposures on malaria infection (level 1).

  The authors have justified their use of an MLM approach based on the ICC of the null model (intercept terms only) being statistically significant (ICC: 0.19, p = 0.046). However, I am reading in another paper - A practical guide to multilevel modelling. Peugh J. Journal of School Psychology 48 (2010) 85 - 112 - that "a non-zero ICC estimate does not necessarily indicate the need for multilevel analyses".

Rather, it is, according to these authors, also dependent on the design effect:

DE = 1 + (n - 1)ICC

Where n is the average number of individuals in each cluster, which, in the paper above is 4.2. Also, in the above paper ICC = 0.19.

Using the numbers from this paper gives me a DE = 1.608.

Peugh goes on to say that "some researchers believe that design effect estimates greater than 2.0 indicate a need for MLM", which, given the values from the malaria paper indicates that an MLM approach was not needed after all.

Were the authors of the malaria paper wrong to have used an MLM approach? Could they have used traditional multivariable logistic regression modelling?

An epidemiology study used mixed-level modelling to investigate the effect of individual and household-level exposures on malaria infection. The authors justified their use of an MLM approach based on the ICC of the null model being statistically significant (ICC: 0.19, p = 0.046). 

However, I am reading another paper - A practical guide to multilevel modelling. Peugh J. Journal of School Psychology 48 (2010) 85 - 112 - where the author states that "a non-zero ICC estimate does not necessarily indicate the need for multilevel analyses".

Rather, it is, according to these authors, dependent on the design effect:

DE = 1 + (n - 1)ICC

Where n is the average number of individuals in each cluster, which, in the paper above is 4.2.

Using the numbers from the malaria paper gives me a DE = 1.608.

Peugh goes on to say that "some researchers believe that design effect estimates greater than 2.0 indicate a need for MLM", which, given the values from the malaria paper indicates that an MLM approach was not needed after all.

I have also read another paper (cited in another post on this website): Nezlek J. "An introduction to multilevel modeling for social and personality psychology" Social and Personality Psychology Compass 2/2 (2008): 842–860, where the authors seem to suggest that if the hierarchal structure of the data indicates clustering, an MLM is justified regardless of the ICC (presumably the design effect then also).

I am confused as I do not have the statistical knowledge/background to make a judgement call myself.

The reason I am nitpicking over this is because, like the malaria study above, I have hierarchal data where the ICC is statistically significant but the design effect is only 1.68. And, if I am justified in using traditional single-level logistic regression then my sample size can be smaller, which would help as I am planning a secondary analysis of existing data where the sample size is fairly limited (using an MLM approach, the sample size only has sufficient power to detect a significant difference if the effect size is around 0.7, for a medium effect size (0.5) the power of the study, given an MLM approach, is only 50%).

So, should I be considering the ICC, the DE, or both in my decision-making?

I am reading a paper that is using mixed-level modelling to investigate the effect of individual (level 1) and household-level (level 2) exposures on malaria infection (level 1).

The authors have justified their use of an MLM approach based on the ICC of the null model (intercept terms only) being statistically significant (ICC: 0.19, p = 0.046). However, I am reading in another paper - A practical guide to multilevel modelling. Peugh J. Journal of School Psychology 48 (2010) 85 - 112 - that "a non-zero ICC estimate does not necessarily indicate the need for multilevel analyses".

Rather, it is, according to these authors, also dependent on the design effect:

DE = 1 + (n - 1)ICC

Where n is the average number of individuals in each cluster, which, in the paper above is 4.2. Also, in the above paper ICC = 0.19.

Using the numbers from this paper gives me a DE = 1.608.

Peugh goes on to say that "some researchers believe that design effect estimates greater than 2.0 indicate a need for MLM", which, given the values from the malaria paper indicates that an MLM approach was not needed after all.

Were the authors of the malaria paper wrong to have used an MLM approach? Could they have used traditional multivariable logistic regression modelling?

I am very curious because I am planning a secondary analysis of a similar dataset for my own research and the need for (or otherwise) an MLM approach affects if my study is going to have sufficient power to detect a significant difference, or not. It may even affect whether I choose this project or not!

I am reading a paper that is using mixed-level modelling to investigate the effect of individual (level 1) and household-level (level 2) exposures on malaria infection (level 1).

The authors have justified their use of an MLM approach based on the ICC of the null model (intercept terms only) being statistically significant (ICC: 0.19, p = 0.046). However, I am reading in another paper - A practical guide to multilevel modelling. Peugh J. Journal of School Psychology 48 (2010) 85 - 112 - that "a non-zero ICC estimate does not necessarily indicate the need for multilevel analyses".

Rather, it is, according to these authors, also dependent on the design effect:

DE = 1 + (n - 1)ICC

Where n is the average number of individuals in each cluster, which, in the paper above is 4.2. Also, in the above paper ICC = 0.19.

Using the numbers from this paper gives me a DE = 1.608.

Peugh goes on to say that "some researchers believe that design effect estimates greater than 2.0 indicate a need for MLM", which, given the values from the malaria paper indicates that an MLM approach was not needed after all.

Were the authors of the malaria paper wrong to have used an MLM approach? Could they have used traditional multivariable logistic regression modelling?