0
$\begingroup$

I am running an analysis of data with multiple observations per person. I assumed that therefore a mixed model analysis would be appropriate.

Back then in my statistics class, we were told that hierarchical modeling is necessary when the ICC of the null model is > .05. However, we were not told what to do when the ICC is lower.

For my data, I specified a null model with only a random intercept. The ICC was 0. I also specified some more complex models (adding fixed and/or random effects) which all led to ICCs > .05.

Where do I go from here? Is it legit to just compute a simple linear regression, as is recommended here? If not, how would I explain that I did a mixed model analysis despite the low ICC?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
1
$\begingroup$

In theory yes, a "low" ICC (below .05 is a common but arbitrary "rule of thumb") means that there is very little variance at the higher level and that therefore random effects are not necessary - so you would just run a "normal" OLS model.

However, in your situation I think something else might be going on. In truly "nested" data - the ICC of a "null" model should (almost) never be EXACTLY zero. Maybe you are just rounding and the ICC was close to zero, but that's still odd in a situation where you have observations clustered within people. In models where people are nested within "groups" (schools, hospitals etc.) it's not uncommon to find very low ICCs (say .02 or something), but when you have observations nested within people ICCs tend to be higher, just because there is a strong likelihood that the responses of "Steve at time 1" and "Steve at time 2" are just going to tend to be way more similar than "Steve at time 1" and "Bob at time 1," which is what the ICC is picking up. So I wonder if the structure of your dataset doesn't work the way you think it does (or maybe you specified the wrong variable as the "clustering variable" - which in your case should be a person-level ID of some kind).

Also, it should be noted that when you add predictors to the model the interpretation of the ICC changes. In a null model the ICC represents the proportion of variance in the dependent variable that is at the second level, but in a model with predictors the ICC you get is the "conditional" ICC - the proportion of the UNEXPLAINED variance in the dependent variable (that is, the variance that remains after accounting for the predictors) that is at the higher level. The fact that you started with an ICC of (near) zero but the ICC became greater than .05 when you added variables is very odd, and is another indication to me that something may be going wrong with the way you are thinking about the different levels of your dataset.

Improve this answer
$\endgroup$
2
  • $\begingroup$ That is very interesting. I am quite confident that my data structure is correct. I have my observations nested in participants (indicated by participant_id). From the same dataset, with other analyses, the ICC is close to zero but not exactly zero. So it might actually be a rounding problem. But I will further investigate this. Thank you for your reply. I will re-check my method of computing the ICC and my data itself. $\endgroup$
    – Max J.
    Commented Feb 3, 2021 at 17:04
  • $\begingroup$ I found out that there was a mistake for computing the ICC when there are fixed effects specified. With another package, I found that the ICC was actually very close to zero (not exactly zero, though). $\endgroup$
    – Max J.
    Commented Feb 4, 2021 at 14:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.