5
$\begingroup$

I know my two samples have unequal variance. Should I consider this in my calculation of the effect size? I'm using Cohen's d, which Wikipedia defines as

$d = \frac{\bar{x}_1 - \bar{x}_2}{s}$

with

$s = \sqrt{\frac{(n_1-1)s^2_1 + (n_2-1)s^2_2}{n_1+n_2 - 2}}$

However, I noticed that when performing the t-test assuming unequal variance, a different estimation of the variance is used (instead of the pooled variance). Should I adapt the Cohen's d formula as well?

$\endgroup$
2
2
$\begingroup$

When doing t tests, we divide the estimated mean difference with standard error \sqrt{Var}/\sqrt{N}. Whereas in Cohen's d, we divide the mean difference with pooled standard deviation. They are usually different.

$\endgroup$
0
$\begingroup$

With unequal variances (where I find that $\frac{S^2_{bigger}}{S^2_{smaller}} \le 1.5$, this would be a moderate difference between variances according to Cohen, 1988) I give a range for d where I divide the difference of the means 1) by the bigger (lower bound for d) and 2) by the smaller (upper bound for d) standard deviation.

I should add that this strategy is purely descriptive and only serves the purpose of helping to interpret the size of the difference that may be there. As far as I know there is no statistical-mathematical argument for this strategy.
(I find it helpful though in cases where it seems plausible from a contextual perspective, say, the difference in well-being between control group and treatment group: even if you absolutely should interpret the differences in variability, one could say, that the relative difference between the means seems quite big, assuming the treatment could be modified in a way that it produces more similar outcomes, but moderate at best considering the seemingly very heterogeneous effects of the treatment.)

$\endgroup$
0
$\begingroup$

There's a recent article by Delacre et al recommending Hedge's g* based on the average SD of the two groups for unequal variance samples. You may check them out.

http://daniellakens.blogspot.com/2015/01/always-use-welchs-t-test-instead-of.html

https://psyarxiv.com/tu6mp/

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.