Effect size: use standard deviation or standard deviation of the differences?

Question

Consider the effect size for a test between two independent samples.

I thought the difference in means compared to the standard deviation, as in this formulas following Cohen

but a recent article in The American Statistician by Demidenko seems to use the standard deviation of the difference, i.e. .

"For example, a widely used effect size of 0.5 means that the proportion of treated patients who do not improve will be roughly 30% and the proportion who do improve will be 70% (D-value = Φ( − 0.5) ≃ 0.3)." This seems only possible if the effect size is calculated with that extra factor of the square root of two, meaning the effect is relative to the standard deviation of the differences between cases, rather than the standard deviation of the cases.

What have I missed here? Are these alternative definitions of effect size?

Answer 1

There are alternative definitions, and I think that which one is appropriate depends on what you want to know.

If you're doing a trial for (say, asthma treatment), then you might have some people who have placebo, and some who have control (regular trial), or you could do a cross-over trial, in which some people start with treatment and then switch to control, and others start with control and then switch to treatment. In this case, you want the effect size to be the expected effect of the treatment, and you want it to be the same, regardless of the type of trial you did (because the effectiveness of the medicine was the same). You use the standard deviation of the scores (of the control group?). If you have paired data your standard error will be much smaller, but the effect size should be the same.

But if you're interested in the amount of change that a person might experience, relative to how much change (not absolute level) that others experience, you should use the difference.

Answer 2

@Jeremy_Miles is correct that Demidenko is using an alternate definition. He refers to an earlier derivation by Browne. I looked up this article today. Browne notes that for an effect size of 0.5 P(X>Y) is .64 [not .70 as Demidenko has it with the alterative definition of effect size] because he is assuming the Cohen formula and so P(X>Y) = -- i.e. that square root of two factor is AFTER the effect size calculation, not before, as Demidenko has it.