3
$\begingroup$

I would like to know if it is possible to statistically test for difference between effect sizes (Hedges g in this case)

For example, let's say I have run a random effects meta-analysis on $20$ studies. In $10$ of these studies the treatment is solely on women and in $10$ the treatment is solely on men. Subgroup analyses show that the summary effect size for women is $g = 0.7,\, 95\% \text{ CI } [0.65-0.75]$ and for men it is $ g = 0.6 \,, 95\% \text{ CI } [0.55-0.65 ] $.

Is there a valid way to test if the effect is stronger in women compared to men?

$\endgroup$
  • 1
    $\begingroup$ You could certainly bootstrap. I don't know if there is another way. $\endgroup$ – Peter Flom - Reinstate Monica May 4 '17 at 12:42
  • 1
    $\begingroup$ The answer is yes, it is called meta-regression. Do you want to do some research on that and then edit your post if you cannot achieve what you desire? $\endgroup$ – mdewey May 4 '17 at 19:21
  • 1
    $\begingroup$ If you are using R, see here: metafor-project.org/doku.php/… You can also easily compute the Wald-type test by hand. $\endgroup$ – Wolfgang May 4 '17 at 20:16
1
$\begingroup$

Yes, this is a rather standard moderator analysis using the analog to the ANOVA procedure originally developed by Hedges' and Olkin. This method is similar to a one-way ANOVA or in this case a t-test but produces a Q statistic that is chi-square distributed. This can be done in R using the metafor package. For Stata, SPSS, or SAS, there are macros http://mason.gmu.edu/~dwilsonb/home.html (note that I am the author of these macros).

Note that your mean effects size may be a bit different under this model as the default is to use a common estimate of the random effects variance component (tau^2). I believe that metafor has an option for estimating separate tau^2 for each group. It is also worth noting that the Cochrane Handbook simply recommends examining the 95% confidence intervals to see if they overlap. While reasonable, this will not always agree with the Q-test, particularly if the standard errors for the two means are substantially different.

$\endgroup$
  • 1
    $\begingroup$ The link in @wolfgang's comment above explains how to estimate with different amounts of heterogeneity using metafor. $\endgroup$ – mdewey May 5 '17 at 10:58
  • 1
    $\begingroup$ Meta-regression and the analog to the ANOVA will produce the same result just different output. $\endgroup$ – dbwilson May 5 '17 at 16:25

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.