19
$\begingroup$

For an effect size analysis, I am noticing that there are differences between Cohen's d, Hedges's g and Hedges' g*.

  • Are these three metrics normally very similar?
  • What would be a case where they would produce different results?
  • Also is it a matter of preference which I use or report with?
$\endgroup$
18
$\begingroup$

Both Cohen's d and Hedges' g pool variances on the assumption of equal population variances, but g pools using n - 1 for each sample instead of n, which provides a better estimate, especially the smaller the sample sizes. Both d and g are somewhat positively biased, but only negligibly for moderate or larger sample sizes. The bias is reduced using g*. The d by Glass does not assume equal variances, so it uses the sd of a control group or baseline comparison group as the standardizer for the difference between the two means.

These effect sizes and Cliff's and other nonparametric effect sizes are discussed in detail in my book:

Grissom, R. J., & Kim, J, J. (2005). Effect sizes for research: A broad practical approach. Mahwah, NJ: Erlbaum.

$\endgroup$
8
$\begingroup$

To my understanding, Hedges's g is a somewhat more accurate version of Cohen's d (with pooled SD) in that we add a correction factor for small sample. Both measures generally agree when the homoscedasticity assumption is not violated, but we may found situations where this is not the case, see e.g. McGrath & Meyer, Psychological Methods 2006, 11(4): 386-401 (pdf). Other papers are listed at the end of my reply.

I generally found that in almost every psychological or biomedical studies, this is the Cohen's d that is reported; this probably stands from the well-known rule of thumb for interpreting its magnitude (Cohen, 1988). I don't know about any recent paper considering Hedges's g (or Cliff delta as a non-parametric alternative). Bruce Thompson has a revised version of the APA section on effect size.

Googling about Monte Carlo studies around effect size measures, I found this paper which might be interesting (I only read the abstract and the simulation setup): Robust Confidence Intervals for Effect Sizes: A Comparative Study of Cohen’s d and Cliff’s Delta Under Non-normality and Heterogeneous Variances (pdf).

About your 2nd comment, the MBESS R package includes various utilities for ES calculation (e.g., smd and related functions).

Other references

  1. Zakzanis, K.K. (2001). Statistics to tell the truth, the whole truth, and nothing but the truth: Formulae, illustrative numerical examples, and heuristic interpretation of effect size analyses for neuropsychological researchers. Archives of Clinical Neuropsychology, 16(7), 653-667. (pdf)
  2. Durlak, J.A. (2009). How to Select, Calculate, and Interpret Effect Sizes. Journal of Pediatric Psychology (pdf)
$\endgroup$
  • 2
    $\begingroup$ An anonymous user wanted to add the following definition of homoscedasticity for those who might be unfamiliar w/ the term: "a property of a set of random variables where each variable has the same finite variance". $\endgroup$ – gung Apr 19 '13 at 16:20
5
$\begingroup$

It seems when people say Cohen's d they mostly mean:

$$d = \frac{\bar{x}_1 - \bar{x}_2}{s}$$

Where $s$ is the pooled standard deviation,

$$s = \sqrt{\frac{\sum(x_1 - \bar{x}_1)^2 + (x_2 - \bar{x}_2)^2}{n_1 + n_2 - 2}}$$

There are other estimators for the pooled standard deviation, probably the most common apart from the above being:

$$s^* = \sqrt{\frac{\sum(x_1 - \bar{x}_1)^2 + (x_2 - \bar{x}_2)^2}{n_1 + n_2}}$$

Notation here is remarkably inconsistent, but sometimes people say that the the $s^*$ (i.e., the $n_1 + n_2$ version) version is called Cohen's $d$, and reserve the name Hedge's $g$ for the version that uses $s$ (i.e., with Bessel’s correction, the n1+n2−2 version). This is a bit weird as Cohen outlined both estimators for the pooled standard deviation (e.g., $s$ version on p. 67, Cohen, 1977) before Hedges wrote about them (Hedges, 1981).

Other times Hedge's g is reserved to refer to either of the bias corrected versions of a standardised mean difference that Hedges developed. Hedges (1981) showed that Cohen's d was upwardly biased (i.e., its expected value is higher than the true population parameter value), especially in small samples, and proposed a correction factor to correct for Cohen's d's bias:

Hedges's g (the unbiased estimator):

$$g = d * (\frac{\Gamma(df/2)}{\sqrt{df/2 \,}\,\Gamma((df-1)/2)})$$ Where $df = n_1 + n_2 -2$ for an independent groups design, and $\Gamma$ is the gamma function. (originally Hedges 1981, this version developed from Hedges and Olkin 1985, p. 104)

However, this correction factor is fairly computationally complex, so Hedges also provided a computationally trivial approximation that, while still slightly biased, is fine for almost all conceivable purposes:

Hedges' $g^*$ (the computationally trivial approximation):

$$ g^* = d*(1 - \frac{3}{4(df) - 1})$$ Where $df = n_1 + n_2 -2$ for an independent groups design.

(Originally from Hedges, 1981, this version from Borenstein, Hedges, Higgins, & Rothstein, 2011, p. 27)

But, as for what people mean when they say Cohen's d vs. Hedges' g vs. g*, people seem to refer to any of these three estimators as Hedge's g or Cohen's d interchangeably, although I've never seen someone write "$g^*$" in a non-methodology/stats research paper. If someone says "unbiased Cohen's d", you're just going to have to take your best guess at either of the last two (and I think there might even be another approximation that has been used for Hedge's $g^*$ too!).

They are all virtually identical if $n > 20$ or so, and all can be interpreted in the same way. For all practical purposes, unless you're dealing with really small sample sizes, it probably doesn't matter which you use (although if you can pick, you may as well use the one that I've called Hedges' g, as it is unbiased).

References:

Borenstein, M., Hedges, L. V., Higgins, J. P., & Rothstein, H. R. (2011). Introduction to Meta-Analysis. West Sussex, United Kingdom: John Wiley & Sons.

Cohen, J. (1977). Statistical power analysis for the behavioral sciences (2nd ed.). Hillsdale, NJ, US: Lawrence Erlbaum Associates, Inc.

Hedges, L. V. (1981). Distribution Theory for Glass's Estimator of Effect size and Related Estimators. Journal of Educational Statistics, 6(2), 107-128. doi:10.3102/10769986006002107

Hedges L. V., Olkin I. (1985). Statistical methods for meta-analysis. San Diego, CA: Academic Press

$\endgroup$
3
$\begingroup$

If you're just trying to understand the basic meaning of Hedges' g, as I am, you might also find this helpful:

The magnitude of Hedges’ g may be interpreted using Cohen's (1988 [2]) convention as small (0.2), medium (0.5), and large (0.8). [1]

Their definition is short and clear:

Hedges’ g is a variation of Cohen's d that corrects for biases due to small sample sizes (Hedges & Olkin, 1985). [1] footnote

I would appreciate statistics experts editing this to add any important caveats to the small (0.2) medium (0.5) and large (0.8) claim, to help nonexperts avoid misinterpreting Hedges' g numbers used in social science and psychology research.

[1] http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2848393/ The Effect of Mindfulness-Based Therapy on Anxiety and Depression: A Meta-Analytic Review Stefan G. Hofmann, Alice T. Sawyer, Ashley A. Witt, and Diana Oh. J Consult Clin Psychol. 2010 April; 78(2): 169–183. doi: 10.1037/a0018555

[2] Cohen J. Statistical power analysis for the behavioral sciences. 2nd ed. Erlbaum; Hillsdale, NJ: 1988 (cited in [1])

$\endgroup$
  • 4
    $\begingroup$ +1. Re: small-medium-large, as a 1st pass, if you have no relevant knowledge or context whatsoever, these 't-shirt sizes' are OK, but in reality, what is a small or large effect will vary by discipline or topic. Moreover, just because an effect is 'large' doesn't necessarily mean it's practically important or theoretically meaningful. $\endgroup$ – gung Aug 10 '13 at 13:07
1
$\begingroup$

The other posters have covered the issue of similarities and differences between g and d. Just to add to this, some scholars do feel that the effect size values offered by Cohen are far too generous leading to overinterpretation of weak effects. They are also not tied to r leading to the possibility scholars may convert back and forth to obtain more favorably interpretable effect sizes. Ferguson (2009, Professional Psychology: Research and PRactice) suggested using the following values for interpretation for g:

.41, as the recommended minimum for "practical significance." 1.15, moderate effect 2.70, strong effect

These are obviously more rigorous/difficult to achieve and not many social science experiments are going to get to strong effects...which is probably how it should be.

$\endgroup$
0
$\begingroup$

Bruce Thompson did warn about using Cohen's (0.2) as small (0.5) as medium and (0.8) as large. Cohen never meant for these to be used as rigid interpretations. All effect sizes must be interpreted based on the context of the related literature. If you are analyzing the related effect sizes reported on your topic and they are (0.1) (0.3) (0.24) and you produce an effect of (0.4) then that may be "large". Conversely, if all the related literature has effects of (0.5) (0.6) (0.7) and you have the effect of (0.4) it may be considered small. I know this is a trivial example but imperatively important. I believe Thompson once stated in a paper, "We would merely be stupid in a different metric" when comparing interpretations of effect sizes to how social scientists were interpreting p values at the time.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.