Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It only takes a minute to sign up.
Sign up to join this community
I know my two samples have unequal variance. Should I consider this in my calculation of the effect size? I'm using Cohen's d, which Wikipedia defines as
$d = \frac{\bar{x}_1 - \bar{x}_2}{s}$
with
$s = \sqrt{\frac{(n_1-1)s^2_1 + (n_2-1)s^2_2}{n_1+n_2 - 2}}$
However, I noticed that when performing the t-test assuming unequal variance, a different estimation of the variance is used (instead of the pooled variance). Should I adapt the Cohen's d formula as well?
When doing t tests, we divide the estimated mean difference with standard error \sqrt{Var}/\sqrt{N}. Whereas in Cohen's d, we divide the mean difference with pooled standard deviation. They are usually different.
With unequal variances (where I find that $\frac{S^2_{bigger}}{S^2_{smaller}} \le 1.5$, this would be a moderate difference between variances according to Cohen, 1988) I give a range for d where I divide the difference of the means 1) by the bigger (lower bound for d) and 2) by the smaller (upper bound for d) standard deviation.
I should add that this strategy is purely descriptive and only serves the purpose of helping to interpret the size of the difference that may be there. As far as I know there is no statistical-mathematical argument for this strategy.
(I find it helpful though in cases where it seems plausible from a contextual perspective, say, the difference in well-being between control group and treatment group: even if you absolutely should interpret the differences in variability, one could say, that the relative difference between the means seems quite big, assuming the treatment could be modified in a way that it produces more similar outcomes, but moderate at best considering the seemingly very heterogeneous effects of the treatment.)
There's a recent article by Delacre et al recommending Hedge's g* based on the average SD of the two groups for unequal variance samples. You may check them out.
http://daniellakens.blogspot.com/2015/01/always-use-welchs-t-test-instead-of.html
