Why is it important to correct for correlated (dependent) measures when reporting Cohen's d?

Question

I am calculating within-group effect sizes from pre-test to post-test. Cohen's d_av reports this effect size as a proportion of the average standard deviation (Lakens, 2013):

Meanwhile, Cohen's d_rm corrects for the fact that the pre-test and post-test measures are correlated (i.e., dependent):

Some argue that d_rm should be used instead of d_av, because pre-test and post-test scores are not independent of one another (Cuijpers et al., 2017). My question is this: Why is the fact that these measures aren't independent a problem? In other words, why should one adjust the effect size estimate based on the correlation of the pre-test and post-test measures?

Consider these two simulated effects:

In the correlated example (left), the correlation between pre-test and post-test measures is .9, d_av is 1, and d_rm is .6. In the uncorrelated example (right), the correlation is .1, d_av is still 1, and d_rm is also 1. In both examples, the SD at pre-test is 1 and the SD at post-test is 2. Clearly, the average person in the correlated example improved the same amount as the average person in the uncorrelated example. So why "correct" the effect size?

Answer 1

This question essentially boils down to the difference between the effect size for a matched-pairs t-test and a 2 independent samples t-test (the former would be the correlated version in your example above). Simply put, these are different statistical tests asking different research questions under different assumptions on the data.

For the matched-pairs approach, the bivariate data can be reasonably transformed into a univariate data set: the gain or difference scores for each individual. Thus, the question becomes a measure of how much each individual respondent gained (as opposed to where they actually ended up). Thus, the effect size is given by $$d = \frac{\overline{d}}{s_d}$$ where $\overline{d}$ is the average gain score (which is mathematically equivalent to the difference in the means) and $s_d$ is the standard deviation of the difference scores. (I didn't work it out, but I'm pretty sure this is equivalent to the formula you've provided in the OP.)

For the 2 independent samples t-test, the effect size is calculated using the pooled standard deviation $$d = \frac{\overline{x}_1-\overline{x}_2}{s_\text{pooled}}$$ This will not give the exact same value of $d_\text{av}$ above, but it will be reasonably close. This is just a measure of how far part averages are in terms of the best measure of "communal" spread for the two groups, i.e., how many standard deviations away are the two means.

Happy to clarify more if needed.