# Effect size measure for pretest-posttest design

I recently conducted an intervention study with random assignment to treatment and control conditions. Participants completed a pretest before the intervention and a posttest after. I analyzed test performance using a 2 (time: pretest or posttest) * 2 (condition: treatment or control) ANOVA. After finding a time * condition interaction, I conducted post-hoc t-tests comparing pretest to posttest within each condition. Together with these t-test results, I reported an effect size within each condition, which I called Cohen's d (hopefully accurately) and calculated as (mean(post) - mean(pre))/sd(pre).

After submitting the results to a journal, I received a reviewer comment saying that Cohen's d does not account for the correlation between pretest and posttest, which can result in an inflated estimate. The reviewer said there is a more correct formula which does not have this problem, but didn't say what the formula was.

My questions: (1) Is it true that Cohen's d, as I calculated it, is not correct in this situation? (2) If so, what is a more correct effect size computation I could use, ideally with a citation?

## 1 Answer

The effect size is correct but the standard error and confidence interval for the effect size must take into account the pre-post correlation. The equation for the standard error of the pre-post Cohen's $$d$$ is: $$se = \sqrt{ \frac{2\left(1-r\right)}{n}+\frac{d^2}{2n}} .$$ That said, I don't think these are the effect sizes you really want and they are misleading. If you have a statistically significant effect in the treatment group but you don't have a statistically significant effect in the control group, that does not mean that the difference is significant. For that, you must look at the interaction. A more meaningful effect size would be of the difference-in-differences. This effect size would reflect the effect of interest in a single number.

• Thanks for your reply. How would I go from the SE formula that you provided to calculating the effect size? (I have to disagree that these aren't the effect sizes I want. The difference-in-differences is important for testing whether there is a treatment effect compared to control, but it's also important to assess how beneficial the treatment is. I used an active control, which might itself be beneficial or harmful, so the beneficial effect of treatment is not simply equivalent to the difference between treatment and control. I can't see any way to answer both questions with a single number.) Commented Oct 4, 2019 at 13:40
• You calculate the effect size as you described, the post-test mean minus the pre-test mean divided by the pre-test SD. I would still argue that this effect size has limited interpretability. With your design, I do not see any realistic way to estimate the full effect of the treatment without a no treatment control. With these effect sizes, you are assuming that individuals would not have changed over time. If that were a reasonable assumption, then why even have a control group in the first place? Commented Oct 6, 2019 at 15:34
• I see your point about interpretability of the effects. I'll have to chew on this for a while. Thanks! Meanwhile, do you happen to know a citation (e.g. textbook or article) that contains the formula you provided? Commented Oct 8, 2019 at 13:03
• Becker, B. J. (1988). Synthesizing standardized mean‐change measures. British Journal of Mathematical and Statistical Psychology, 41(2), 257-278. Also, Lipsey and Wilson (2001) Practical Meta-analysis. Commented Oct 9, 2019 at 23:41
• Can you please elaborate on this line in your answer or in the comment -- "A more meaningful effect size would be of the difference-in-differences?" Does this mean we should calculate effect size based on the (treatment_post - treatment_pre) and (control_post - control_pre) as the two groups? Commented Feb 5, 2023 at 2:30