2
$\begingroup$

I am planning a meta-analysis. I read some books (e.g., Cumming, 2012) and papers (e.g., Johnson & Eagly, 2000). I learned that standardized measure of effect size like d and r are sensitive to resaerch design so that standardized effect size for within-subject design are usually larger than those for between-subject design in spite of the fact that the unstandardized effect sizes of both design are exactly the same (e.g., Baguley, 2009). For my meta-analysis, I have some studies with a between-subject design and other with a within-subject design. I know that I can correct standardized effect size for research design (e.g., Cumming, 2012, p. 291; Olejnik & Algina, 2000).

My question is : Do I need to do more than correcting for research design in order to combine my effect sizes? A similar question was already asked by Jemma (A meta-regression with effect sizes from variously designed studies) and mdewey suggested to convert the effect size into the same scale. I am ok with that and correcting for research design is a part of the job. But is it necessary to take into account in the computation of the combined effect size that some studies followed a within-subject design or a between-subject design?

Edit: I maybe found something consistent with mdewey's answer to Jemma in Borenstein et al. (2009, p. 30). They stated that:

a single systematic review can include studies that used independent groups and also studies that used matched groups. From a statistical perspective the effect size (d or g) has the same meaning regardless of the study design. Therefore, we can compute the effect size and variance from each study using the appropriate formula, and then include all studies in the same analysis. While there are no technical barriers to using studies with different designs in the same analysis, there may be a concern that these studies could differ in substantive ways as well (see Chapter 40)

Does someone else have some relevant references allowing to corroborate this statement?

Thanks a lot in advance

$\endgroup$
  • $\begingroup$ Can you clarify how your question differs from the one you cite? $\endgroup$ – mdewey Jan 8 '17 at 15:20
  • $\begingroup$ Thanks for your comment. It is a very similar question. However, even if I agree with your answer to this question that converting effect sizes into the same scale is the first step, I am still wondering wether other steps are required. $\endgroup$ – Psychokwak Jan 10 '17 at 10:28
  • 1
    $\begingroup$ I might add a moderator variable with levels equal to the number of different study designs to take account of any overall differences. If they were large I might stratify the analysis by study design but for me that would be a last resort. $\endgroup$ – mdewey Jan 10 '17 at 10:59
  • $\begingroup$ Perhaps there is a scale where this just does not matter - e.g. if different research designs measured the effect of a drug on blood pressure compared to a placebo, then staying on the mmHg scale instead of going to a BP/SD of BP scale avoids the whole issue. $\endgroup$ – Björn Feb 21 at 5:02
1
$\begingroup$

Morris and DeShon have a paper that answers this question:

Morris, S. B., & DeShon, R. P. (2002). Combining effect size estimates in meta-analysis with repeated measures and independent-groups designs. Psychological methods, 7(1), 105-125.

They recommend that: 1) You convert the effect sizes into a common metric, and 2) You assess design as a moderator.

If design is a significant moderator then you need to either meta-analyse the within-subjects and between-subjects design studies separately, or use multiple-meta regression where all moderators are assessed simultaneously (i.e., so that when you assess each moderator, the rest are controlled for).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.