I am currently running a three-level meta-analysis, due to the fact that I have multiple dependent effect sizes in many studies. I do not know the actual dependencies between effect sizes, however, so when there are dependencies, they are assumed to be $\tau^2_{(3)}$, per Cheung (2015, Ch. 6) and Konstantopoulos (2011). I am using the rma.mv() function in the metafor package to do this and employing a meta-regression approach, such that my effect sizes es (in my meta-analysis, Fisher's Z) are predicted by two moderators, pubyear and subtle. The model looks like this:

mod2.1 <- rma.mv(es ~ pubyear + subtle, 
                 random = ~1|studyid/effectid, 
                 V = var, data = meta)

My main interest is in pubyear. I find that, after adding subtle as an additional covariate, the effect of pubyear gets smaller.

This leads me to ask the question: Is subtle predicted by pubyear? I do not know how I would estimate this, so my question is: How can I model the relationship between two moderators in a meta-regression?

Right now, my meta-analysis shows that the es gets smaller as pubyear increases, so now I want to know: Has the subtlety of measures (subtle) also increased over time (pubyear)? How would I do this?

The intuitive solution would to simply correlate the two with one another. But this does not take into account that one study might report 8 effect sizes, while another study may only report 1. Obviously, the study with 8 effect sizes will have more weight. I plotted this relationship, regardless. You can see that there may be a relationship when doing this weirdly-weighted regression (lm line in blue, loess line in red):

enter image description here

Has anyone run into this before or know how to handle it?

  • 1
    $\begingroup$ I'll take a look more in-depth later in the week if I can find the time, but in the meantime, Lipsey (2003; jstor.org/stable/1049948?seq=1#page_scan_tab_contents) and this CV thread here (stats.stackexchange.com/questions/262326/…) might be able to get you started. $\endgroup$
    – jsakaluk
    May 27, 2017 at 22:12
  • $\begingroup$ Brilliant, thanks. I didn't think about searching for VIF. $\endgroup$
    – Mark White
    May 27, 2017 at 22:14
  • 1
    $\begingroup$ Also worth looking at the variance covariance matrix of your coefficients which is sometimes enlightening. $\endgroup$
    – mdewey
    May 29, 2017 at 13:12

1 Answer 1


So, if I understand things correctly, subtle is a variable at the effect size level and pubyear is a variable at the study level. If you really want to examine if subtle is predicted by pubyear, then this could be done using a standard multilevel model (with random intercepts). Using the nlme package, you can fit this with:

res <- lme(subtle ~ pubyear, random = ~ 1 | studyid, data=meta)

Of course, this assumes a linear relationship, but more complex relationships can also be modeled (but I wouldn't use that loess line as any indication, as that seems like a total overfit in my opinion).

  • 1
    $\begingroup$ Yup, you got it. Thanks, I thought about doing this, too, but wasn't sure. I also thought about doing a WLS where the weights for each data point would be $1/k_j$, where $k$ is the number of effect sizes in study $j$. This would control for the problem that some studies report 15 effect sizes, while others report 1, leading an OLS to be biased. But I think I'll fit an lmer object, because it doesn't ignore the dependencies between the effect sizes within a study. $\endgroup$
    – Mark White
    May 29, 2017 at 21:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.