2
$\begingroup$

I recently conducted an intervention study with random assignment to treatment and control conditions. Participants completed a pretest before the intervention and a posttest after. I analyzed test performance using a 2 (time: pretest or posttest) * 2 (condition: treatment or control) ANOVA. After finding a time * condition interaction, I conducted post-hoc t-tests comparing pretest to posttest within each condition. Together with these t-test results, I reported an effect size within each condition, which I called Cohen's d (hopefully accurately) and calculated as (mean(post) - mean(pre))/sd(pre).

After submitting the results to a journal, I received a reviewer comment saying that Cohen's d does not account for the correlation between pretest and posttest, which can result in an inflated estimate. The reviewer said there is a more correct formula which does not have this problem, but didn't say what the formula was.

My questions: (1) Is it true that Cohen's d, as I calculated it, is not correct in this situation? (2) If so, what is a more correct effect size computation I could use, ideally with a citation?

Improve this question
$\endgroup$

1 Answer 1

Reset to default
2
$\begingroup$

The effect size is correct but the standard error and confidence interval for the effect size must take into account the pre-post correlation. The equation for the standard error of the pre-post Cohen's $d$ is: $$ se = \sqrt{ \frac{2\left(1-r\right)}{n}+\frac{d^2}{2n}} . $$ That said, I don't think these are the effect sizes you really want and they are misleading. If you have a statistically significant effect in the treatment group but you don't have a statistically significant effect in the control group, that does not mean that the difference is significant. For that, you must look at the interaction. A more meaningful effect size would be of the difference-in-differences. This effect size would reflect the effect of interest in a single number.

Improve this answer
$\endgroup$
6
  • $\begingroup$ Thanks for your reply. How would I go from the SE formula that you provided to calculating the effect size? (I have to disagree that these aren't the effect sizes I want. The difference-in-differences is important for testing whether there is a treatment effect compared to control, but it's also important to assess how beneficial the treatment is. I used an active control, which might itself be beneficial or harmful, so the beneficial effect of treatment is not simply equivalent to the difference between treatment and control. I can't see any way to answer both questions with a single number.) $\endgroup$
    – baixiwei
    Commented Oct 4, 2019 at 13:40
  • 1
    $\begingroup$ You calculate the effect size as you described, the post-test mean minus the pre-test mean divided by the pre-test SD. I would still argue that this effect size has limited interpretability. With your design, I do not see any realistic way to estimate the full effect of the treatment without a no treatment control. With these effect sizes, you are assuming that individuals would not have changed over time. If that were a reasonable assumption, then why even have a control group in the first place? $\endgroup$
    – dbwilson
    Commented Oct 6, 2019 at 15:34
  • $\begingroup$ I see your point about interpretability of the effects. I'll have to chew on this for a while. Thanks! Meanwhile, do you happen to know a citation (e.g. textbook or article) that contains the formula you provided? $\endgroup$
    – baixiwei
    Commented Oct 8, 2019 at 13:03
  • 1
    $\begingroup$ Becker, B. J. (1988). Synthesizing standardized mean‐change measures. British Journal of Mathematical and Statistical Psychology, 41(2), 257-278. Also, Lipsey and Wilson (2001) Practical Meta-analysis. $\endgroup$
    – dbwilson
    Commented Oct 9, 2019 at 23:41
  • $\begingroup$ Can you please elaborate on this line in your answer or in the comment -- "A more meaningful effect size would be of the difference-in-differences?" Does this mean we should calculate effect size based on the (treatment_post - treatment_pre) and (control_post - control_pre) as the two groups? $\endgroup$
    – Outsider
    Commented Feb 5, 2023 at 2:30

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.