3
$\begingroup$

I am calculating within-group effect sizes from pre-test to post-test. Cohen's dav reports this effect size as a proportion of the average standard deviation (Lakens, 2013):

enter image description here

Meanwhile, Cohen's drm corrects for the fact that the pre-test and post-test measures are correlated (i.e., dependent):

enter image description here

Some argue that drm should be used instead of dav, because pre-test and post-test scores are not independent of one another (Cuijpers et al., 2017). My question is this: Why is the fact that these measures aren't independent a problem? In other words, why should one adjust the effect size estimate based on the correlation of the pre-test and post-test measures?

Consider these two simulated effects:

enter image description here

In the correlated example (left), the correlation between pre-test and post-test measures is .9, dav is 1, and drm is .6. In the uncorrelated example (right), the correlation is .1, dav is still 1, and drm is also 1. In both examples, the SD at pre-test is 1 and the SD at post-test is 2. Clearly, the average person in the correlated example improved the same amount as the average person in the uncorrelated example. So why "correct" the effect size?

$\endgroup$

1 Answer 1

0
$\begingroup$

This question essentially boils down to the difference between the effect size for a matched-pairs t-test and a 2 independent samples t-test (the former would be the correlated version in your example above). Simply put, these are different statistical tests asking different research questions under different assumptions on the data.

For the matched-pairs approach, the bivariate data can be reasonably transformed into a univariate data set: the gain or difference scores for each individual. Thus, the question becomes a measure of how much each individual respondent gained (as opposed to where they actually ended up). Thus, the effect size is given by $$d = \frac{\overline{d}}{s_d}$$ where $\overline{d}$ is the average gain score (which is mathematically equivalent to the difference in the means) and $s_d$ is the standard deviation of the difference scores. (I didn't work it out, but I'm pretty sure this is equivalent to the formula you've provided in the OP.)

For the 2 independent samples t-test, the effect size is calculated using the pooled standard deviation $$d = \frac{\overline{x}_1-\overline{x}_2}{s_\text{pooled}}$$ This will not give the exact same value of $d_\text{av}$ above, but it will be reasonably close. This is just a measure of how far part averages are in terms of the best measure of "communal" spread for the two groups, i.e., how many standard deviations away are the two means.

Happy to clarify more if needed.

$\endgroup$
1
  • $\begingroup$ Thank you for your response! I believe the first example you give there is d(z). Because the denominator is the standard deviation of the differences, in cases where there is a high correlation between measures it will be quite high and in cases where there is a low correlation between measures it will be quite low. In my example, d(z) for the correlated measures (left) is 1.3, and d(z) for the uncorrelated measures (right) is .7. It's a different measure from d(rm), which is what I use above. $\endgroup$
    – ila
    Apr 15, 2023 at 16:48

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.