I have been using the excellent metafor package for a work-related project and for some reason there's a concept that I can't wrap my head around.

In his guide (https://www.metafor-project.org/doku.php/analyses:konstantopoulos2011?s[]=multilevel) Wolfgang outlines how to calculate the ICC for three-level meta-analytic models based on the following formula:

This can be interpreted as the correlation in "true" effects within a cluster.

In another guide Wolfgang describes how you can calculate a multilevel version of I^2 and also parse this into it's components (between and within) through the following.

100 * model$sigma2 / (sum(model$sigma2) + (model$k-model$p)/sum(diag(P)))

This produces two values (one for within cluster, and a second value for between cluster), and represents the proportion of variance explained due to heterogeneity within each level, respectively.

However, people often refer to ICC's as a measure of the variance explained/accounted for (for example by multiplying by 100 to get the value in percentage units), which confuses me a little when it comes to differences in the interpretation.

My questions are:

  1. Is it correct to say that the ICC represents the similarity between two observations from the same cluster?

  2. What is the difference between the within cluster I^2 value and the ICC? If I^2 value for within cluster represents the proportion of variance explained by heterogeneity, then how is the ICC different when expressed as a %? Is this not also a measure of the proportion of the total variance explained?


1 Answer 1

  1. The ICC in the example/model you linked to (and the correct equation is $ICC = \frac{\sigma^2_1}{\sigma^2_1 + \sigma^2_2}$, that is, we need to plug the variances into the equation, not the SDs) represents the correlation between the underlying true effects from the same cluster.

  2. The 'between-cluster' $I^2$ is indeed quite similar and is given by $I^2 = \frac{\sigma^2_1}{\sigma^2_1 + \sigma^2_2 + \tilde{v}}$, where $\tilde{v}$ can be interpreted as the 'typical' sampling variance. This can be interpreted as the amount of total variance (the sum of the between- and within-cluster heterogeneity and the amount of sampling variability) that is due to between-cluster heterogeneity. One could also interpret this as the correlation between two observed effects from the same cluster (assuming that these two observed effects have sampling variances equal to $\tilde{v}$).

  3. The 'within-cluster' $I^2$ is $I^2 = \frac{\sigma^2_2}{\sigma^2_1 + \sigma^2_2 + \tilde{v}}$. It indicates how much of the total variance is due to within-cluster heterogeneity. This one doesn't lend itself to a correlation interpretation.

  • $\begingroup$ Thank you, Wolfgang. Your response is most appreciated. You are right, I accidentally forgot the squared terms in the equation (my bad)! This has really helped clear things up for me. I have one more question: If I wanted to report the correlation between two outcomes reported from the same study (assuming my model was specified as random= ~ 1| Study/Outcome would I be best to use the between cluster I^2 (i.e., the observed) or the ICC (i.e., the true). Perhaps reporting both is informative? Once again thank you for taking the time out of your day to assist- you are a hero! $\endgroup$ Aug 28, 2021 at 13:28

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.