I have the following problem: I am researching the difference in effects (as measured by standarized mean differences (SMD)) of two intervention types. (let's call them intervention 1 and intervention 2) by pooling the effects of the two different intervention types into pooled effect sizes using meta-analyses. To clarify what I mean: I ran a meta-analysis including 6 studies implementing intervention 1 and a separate analysis including 7 studies implementing intervention 2 and so far I have found that the pooled effect size of intervention 1 is d=0.3 and the pooled effect size of studies implementing intervention 2 is d=0.6. This seems like quite a difference but how do i statistically test whether these effects are significantly different?

If i disregard the meta-analysis and just test whether the mean SMD of studies implementing intervention 2 is significantly larger than studies implementing intervention 1 (t-test) I do not find a significant effect (p=.450). However in this t-test analyses the sample sizes of the individual studies are not taken into account and a SMD of a study with n=10 weighs as heavily as one with n=200. In the meta-analyses, the N of each study obviously does count and it appears that studies implementing intervention 2 with a large sample tend to have a more positive SMD hereby increasing the pooled effect size.

I have been searching the web for an answer to this question but I have yet to find an answer. Would it be viable to conduct an ANCOVA analysis with nominal variable=intervention type (1 or 2), covariate=N and independent variable=SMD?


It's called a meta-regression. That might be all you need. Exactly how you do it depends on your software.

For each study, you have an effect size (smd), and standard error (or variance) associated with that effect size (se or var), and a group variable, with the values 0 and 1. (Or any other two numbers, but 0 and 1 are the most straightforward.

If you're using Stata, then you use the metareg command:

metareg smd group, wsse(se)

If you're using the metafor package in R:

rma(yi=smd, vi=var, data=myData, mods=~group)
| cite | improve this answer | |
  • 1
    $\begingroup$ Also useful for the OP: I have a pretty extensive write-up of exactly this issue on the metafor website. See here: metafor-project.org/doku.php/… $\endgroup$ – Wolfgang Apr 24 '15 at 14:28
  • 1
    $\begingroup$ Another important issue: When comparing interventions against each other like that, this only provide indirect evidence, since the interventions have not been compared directly against each other in a head-to-head comparison. So, the results must be interpreted with caution. $\endgroup$ – Wolfgang Apr 24 '15 at 14:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.