I am meta-analyzing with multiple effect sizes from each study.

The studies included in my meta-analysis measured the effects of participants REPEATDLY with MULTIPLE TESTS. I am trying to deal with dependent effect sizes using three-level meta-analysis.

I see, some people use three level meta-analysis without specifying correlations of effect sizes within clusters. For example, Mike Cheung's answer here (How to best handle subscores in a meta-analysis?), metafor package's example of Konstantopoulos's (2011) analysis (http://www.metafor-project.org/doku.php/analyses:konstantopoulos2011), and so on.

However, Hox's (2010) explanation of "multivariate meta-analysis" states "A serious limitation for multivariate meta-analysis is that the required information on the correlations between the outcome variables is often not available in the publications (p. 223)". Also, metafor package's explanation of Berkey et al. (1998) (http://www.metafor-project.org/doku.php/analyses:berkey1998), indicates that we have to know the covariance and we have to specify the variance-covariance matrix...

My question is: (1) if the studies REPEATEDLY measured same participants, and/or, the same participants measured with different measurements, do I need to know correlations between dependent effect sizes for the three level meta-analysis?

(2) When Do I need to know/assume correlations between dependent effect sizes (Or, covariance structure) for three-level meta-analysis?

Thank you.

Hox, J. J. (2010). Multilevel analysis: techniques and applications (2. ed). New York: Routledge, Taylor & Francis.

  • 1
    $\begingroup$ See my answer to a similar question here: stats.stackexchange.com/questions/327186/… $\endgroup$
    – jsakaluk
    Mar 10, 2018 at 19:23
  • $\begingroup$ I would like to ask you a follow up question, @jskaluk. I found Wolfgang's answer on CV saying that "But this model [i.e., multilevel meta-analysis] still assumes that the sampling errors of the observed outcomes/effects within a study are independent, which is definitely not the case when those outcomes are assessed within the same individuals." (stats.stackexchange.com/questions/166964/…). $\endgroup$ Mar 10, 2018 at 20:29
  • $\begingroup$ Do you think I CANNOT use multilevel modelling to my data set, where sampling errors of the observed effect sizes are not independent because same participants took tests repeatedly and I do know now the correlations? I feel we can model specifying the participant group as another level... Do you think I can do that? $\endgroup$ Mar 10, 2018 at 20:29
  • 1
    $\begingroup$ You should add random effects at the study level and specify the covariances among the sampling errors. $\endgroup$
    – Wolfgang
    Mar 13, 2018 at 19:44
  • 1
    $\begingroup$ Section 6.4 in Cheung (2015) Meta-Analysis: A Structural Equation Modeling Approach discusses how the multivariate and three-level meta-analyses are related. The Google books provide a preview here. $\endgroup$ Mar 22, 2018 at 12:54

1 Answer 1


Whenever the same participants contribute information to multiple estimates (whether it be multiple estimates of the measure over time, multiple effects for different measures, or both), the sampling errors of the estimates are correlated and that should be accounted for by computing the covariances. In addition, one can account for possible dependency in the underlying true effects by fitting a multilevel/multivariate model. That just means that we include random effects, such that the true effects are allowed to be correlated in some form. An example analysis of this is provided here.

A somewhat less common case arises when we have multiple estimates that are clustered within some higher level grouping variable, but the estimates within the groups are based on different sets of participants. In that case, the sampling errors are independent, but there may still be correlation among the underlying true effects. So, we then again want to add random effects that reflect this. An example analysis of this is provided here. As was mentioned in the comments, this type of analysis/model is only appropriate when the sampling errors are uncorrelated.

A common problem with the first case (where sampling errors are correlated) is that the information needed to compute the covariances is not available. Some work has been done to examine whether we can ignore the covariances (i.e., assuming that they are 0) as long as we still allow the underlying true effects to be correlated (again, by adding appropriate random effects to the model). When doing that, the correlations among the sampling errors get subsumed into the correlation among the true effects. Using the example here, you can try this out:

dat <- get(data(dat.berkey1998))
V <- bldiag(lapply(split(dat[,c("v1i", "v2i")], dat$trial), as.matrix))
res1 <- rma.mv(yi, V, mods = ~ outcome - 1, random = ~ outcome | trial, struct="UN", data=dat)
res2 <- rma.mv(yi, diag(V), mods = ~ outcome - 1, random = ~ outcome | trial, struct="UN", data=dat)

Model res1 allows sampling errors to be correlated (the V matrix includes the covariances). In model res2, we use diag(V), so only the diagonal of the V matrix is used. As you will find, the correlation among the true outcomes is 0.6088 in res1 and 0.7752 in res2. Since we (incorrectly) assume that the covariances among the sampling errors are zero in res2, the model tries to compensate for this by increasing the correlation among the underlying true effects.

Model res2 can work okay for making inferences about the fixed effects (i.e., for testing and constructing confidence intervals for the estimated average effects). But we know that the model is misspecified, so it isn't ideal.

A possible remedy is to use cluster-robust inference methods after having fitted a model like res2. For the example above, this would be:

robust(res2, cluster=dat$trial)

The cluster-robust inference approach tries to "fix up" the standard errors of the estimates of the fixed effects. You will see in the example that the SEs are quite a bit larger now. Asymptotically (i.e., when the number of clusters gets large), this approach should provide appropriate SEs (but that's unlikely to be the case in this example, since there are only 5 studies).

  • $\begingroup$ Thank you very much for your detailed explanation, @Wolfgang. Because of being a non-native speaker of English and my limited understanding of statistics, I am having difficulty understanding your explanations. I would appreciate it if I could ask you several more follow-up questions. First, I cannot understand why each effect size (or, trial) has covariance value (i.e., v2i1) in the Berkey et al.'s example (metafor-project.org/doku.php/analyses:berkey1998). Isn't there only one covariance value between PD and AL in each study? $\endgroup$ Mar 25, 2018 at 20:20
  • $\begingroup$ Second, so if I am understanding correctly, Konstantopoulos's (2011) example can be fitted three-level MA without specifying covariances because effect sizes are not correlated by sampling error but by hierarchy. Is that right? If that’s the case, three-level model should not be fitted in cases where participants were repeatedly measured and/or measured with multiple measurements without specifying covariances. Am I understanding correctly? $\endgroup$ Mar 25, 2018 at 22:21
  • $\begingroup$ Third, I am also not sure what exactly res2 is fitting. If the res2 is the example of the case where we fit the model without specifying the covariance but with random effects (i.e. hierarchy structure), we do not even need to put the diagonal of the V matrix, right? (Or, we cannot put diagonal since we would not know…?). What diagonal of the V matrix means here? As far as I understand, robust() can be used without specifying V matrix but just specifying random effect (i.e., hierarchy structure)… right? Or, am I missing something? $\endgroup$ Mar 25, 2018 at 22:24
  • $\begingroup$ 1) There is one covariance per trial. So, for example 0.0030 for trial 1. 2) Correct (at least, we know that the model is misspecified then). 3) res2 shows what happens when we ignore the covariances. Along the diagonal of V are the sampling variances. Those should always be specified. $\endgroup$
    – Wolfgang
    Mar 26, 2018 at 13:37
  • $\begingroup$ >1) Oh, I see. So, the covariances are not the covariances between the PD and AL, but between two values to calculate each effect size (probably between different time points (e.g., pre-test & post-test) or different conditions (e.g., control group & treatment group). >2) and 3) Thank you for clarifying. Now I understand that diag(V) represents the variances for effect sizes. $\endgroup$ Mar 26, 2018 at 17:05

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.