I'm conducting a three-level meta-analysis on correlations using the {metafor} package on R.

I'm quite a newbie when it comes to three-level meta-analyses, so I have two doubts.

1) From each study, I collected effects sizes that indicate:

  • correlations between variable A and variable B (A-B)
  • correlations between variable A and variable C (A-C)

Simplified example of the database with the effect size:

> data

es.id   study.id   z      type
1       Study1     0.10   A-B
2       Study1     0.20   A-B
3       Study1     0.30   A-C
4       Study2     0.15   A-B
5       Study2     0.18   A-C
6       Study3     0.13   A-B
7       Study3     0.10   A-C
8       Study4     0.10   A-B
9       Study4     0.12   A-C
10      Study4     0.09   A-C

I planned to run two separate meta-analyses on this effect sizes (a meta-analysis for A-B correlations and another one for A-C correlations). However, is it possible to compare the two pooled correlation coefficients, namely to say whether A-B correlations are stronger than A-C correlations in the same studies?

Intuitively, I'd run a three-level mixed-effect model of the entire pool of effect sizes (A-B and A-C correlations together), using the "type" of correlation (A-B vs. A-C) as a sort of within-study moderator.

Something like this:

rma.mv(yi, V, data, random = ~ 1 | study.id/es.id, method = "REML", mods = ~ type)

However, I'm not sure this is the best way to proceed and I could not find any study that approached the analysis as I planned to...

2) For my meta-analysis, I also planned to assess publication bias. By looking at the best strategies to adopt with three-level meta-analyses, I decided to use Egg's Test with standard error as a moderator:

egger <- rma.mv(yi, V, mods = ~ se, random = ~1 | study.id/es.id, data)

However, I failed in finding a reliable source concerning the possible correction strategies that can be applied to three-level meta-analyses. I'd like to correct potential publication bias in my study, but most of the online resources describe strategies or procedures that are not easily applicable to rma.mv objects or complex analyses. I would really appreciate it if you could point out some references on how to deal with publication bias on R with {metafor} or other packages.

Thank you everybody for your help!


1 Answer 1

  1. To compare the A-B and A-C correlations, you need to account for the dependency therein, not just by including random effects, but also for the dependency in the sampling errors. The rcalc() function (see here) helps with with computing those and constructing the proper V matrix. Note that you will need to include the B-C correlations in your dataset (since the covariance between $r_{AB}$ and $r_{AC}$ depends on $r_{BC}$). Also, instead of random = ~ 1 | study.id/es.id, you might want to use random = ~ type | study.id, struct="UN".

  2. Including the standard error as a predictor in your model (in addition to type) would be the straightforward generalization of the 'regression test for funnel plot asymmetry'. See:

    Fernández-Castilla, B., Declercq, L., Jamshidi, L., Beretvas, S. N., Onghena, P. & Van den Noortgate, W. (2021). Detecting selection bias in meta-analyses with multiple outcomes: A simulation study. The Journal of Experimental Education, 89(1), 125-144. https://doi.org/10.1080/00220973.2019.1582470

    Rodgers, M. A. & Pustejovsky, J. E. (2021). Evaluating meta-analytic methods to detect selective reporting in the presence of dependent effect sizes. Psychological Methods, 26(2), 141-160. https://doi.org/10.1037/met0000300

  • $\begingroup$ Thank you, you really made my day!! For question (1), do you think that adding a random structure to the model as: ~ type | study.id/es. id (with effect sizes nested in studies) would be an extreme (and/or unuseful) complication to the random structure? As for (2), with "in addition to type" you mean something like this: rma.mv(yi, V, mods = ~ se + type, random = ~ type | study.id, data)? Thank you!! $\endgroup$
    – ternuo
    Commented May 25, 2022 at 21:22
  • $\begingroup$ (1) ~ type | study.id/es. id is not even possible even if you tried. It would also be overparameterized even if rma.mv() would allow this. (2) Yes. And note, you want to use struct="UN", as otherwise it will assume struct="CS" which probably isn't appropriate here. $\endgroup$
    – Wolfgang
    Commented May 26, 2022 at 9:30
  • 1
    $\begingroup$ Thank you for the clarification. Your answers have been extremely useful! $\endgroup$
    – ternuo
    Commented May 26, 2022 at 9:38
  • $\begingroup$ I have read that if correlations come from mixed models, then the slop would be the best effect size indicator. But then, when making a meta analysis of such correlations, would you need to add the error of such slope estimates? The example does not add it. $\endgroup$ Commented Oct 12, 2022 at 8:06

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.