4
$\begingroup$

I found different answers to the question how to calculate the standard error (SE) of Cohen's d.

First formula is (see here, here or here):

$$ SE_d = \sqrt{\frac{n_1 + n_2}{n_1 n_2} + \frac{d^2}{2(n_1+n_2)}} $$

Second formula is (see here): $$SE_d = \sqrt{\left(\frac{n_1 + n_2}{n_1 n_2} + \frac{d^2}{2(n_1+n_2-2)}\right) \left(\frac{n_1 + n_2}{n_1+n_2-2} \right)}$$

Third formula is a slight variation of the first one (see here in the last line of formulae):

$$ SE_d = \sqrt{\frac{n_1 + n_2}{n_1 n_2} + \frac{d^2}{2(n_1+n_2 - 2)}} $$

I know that there is some confusion on how to calculate Cohen'd itself. Cohen's d is defined as $d = \frac{\bar{x_1} - \bar{x_2}}{sd_{pooled}}$ but the pooled standard deviation is defined in two different ways, i.e. $sd_{pooled} = \sqrt{\frac{(n_1 - 1)S_1^2 + (n_2 - 1)S_2^2}{n_1 + n_2}}$ and $sd_{pooled} = \sqrt{\frac{(n_1 - 1)S_1^2 + (n_2 - 1)S_2^2}{n_1 + n_2 - 2}}$ (see here). Does the formula for SE change depending on how $sd_{pooled}$ is defined? Or, if we use always the same formula for SE of Cohen's d: Which of the fomulae above is it?

$\endgroup$
1
  • 1
    $\begingroup$ @user248711 See this review for an examination of all these questions. $\endgroup$ Commented Feb 12, 2022 at 21:07

1 Answer 1

4
$\begingroup$

The statistic Cohen's d follows a scaled non-central t-distribution.

This statistic is the difference of the mean divided by an estimate of the sample standard deviation of the data:

$$d = \frac{\bar{x}_1-\bar{x}_2}{\hat{\sigma}}$$

It is used in power analysis and relates to the t-statistic (which is used in significance testing)

$$d = n^{-0.5} t $$

This factor $n$ is computed as $n=\frac{n_1 n_2}{n_1+n_2}$

The difference is that

  • to compute $d$ we divide by the standard deviation to the data
  • and for $t$ we divide by the standard error of the means

(and these differ by a factor $\sqrt{n}$)

Confidence interval based on normal approximation of non-central t-distribution

The articles that you mention relate to the article Larry V. Hedges 1981 "Distribution Theory for Glass's Estimator of Effect Size and Related Estimators"

There they give a large sample approximation of Cohen's d as a normal distribution with the mean equal to $d$ and the variance equal to $$\frac{n_1 + n_2}{n_1n_2} + \frac{d^2}{2(n_1+n_2)}$$

These expressions stem from the mean and variance of the non-central t-distribution. For the variance we have:

$$\begin{array}{crl} \text{Var}(t) &=& \frac{\nu(1+\mu^2)}{\nu-2} - \frac{\mu^2 \nu}{2} \left(\frac{\Gamma((\nu-1)/2)}{\Gamma(\nu/2)}\right)^2 \\ &\approx& \frac{\nu(1+\mu^2)}{\nu-2} - \frac{\mu^2 \nu}{2} \left(1- \frac{3}{4\nu-1} \right)^{-2} \end{array} $$

Where $\nu = n_1+n_2-2$ and $\mu = d \sqrt{\frac{n_1n_2}{n_1+n_2}}$. For cohen's d this is multiplied with ${\frac{n_1+n_2}{n_1n_2}}$

$$\text{Var}(d) = \frac{n_1+n_2}{n_1n_2} \frac{\nu}{\nu-2} + d^2 \left( \frac{\nu}{\nu-2} -\frac{1}{(1-3/(4\nu-1))^2} \right)$$

The variations in the three formula's that you mention are due to differences in simplifications like $\nu/(\nu-2) \approx 1$ or $\nu = n_1+n_2-2 \approx n_1+n_2$.

In the most simple terms

$$\frac{\nu}{\nu-2} = 1 + \frac{2}{\nu-2} \approx 1$$

and (using a Laurent Series)

$$\frac{\nu}{\nu-2} -\frac{1}{(1-3/(4\nu-1))^2} = \frac{1}{2\nu} + \frac{31}{16\nu^3} + \frac{43}{8\nu^3} + \dots \approx \frac{1}{2\nu} \approx \frac{1}{2(n_1 + n_2)} $$

Which will give

$$\text{Var}(d) \approx \frac{n_1+n_2}{n_1n_2} + d^2\frac{1}{2(n_1+n_2)} $$

Confidence interval based on computation

If you would like to compute the confidence interval more exactly then you could compute those values of the non-central t-distribution for which the observed statistic is an outlier.

Example code:

### input: observed d and sample sizes n1 n2
d_obs = 0.1
n1 = 5
n2 = 5

### computing scale factor n and degrees of freedom
n  = n1*n2/(n1+n2)
nu = n1+n2-2


### a suitable grid 'ds' for a grid search
### based on 
var_est <- n^-1 + d_obs^2/2/nu
ds <- seq(d_obs-4*var_est^0.5,d_obs+4*var_est^0.5,var_est^0.5/10^4)


### boundaries based on limits of t-distributions with ncp parameter 
### for which the observed d will be in the 2.5% left or right tail
upper <- min(ds[which(pt(d_obs*sqrt(n),nu,ds*sqrt(n))<0.025)])*sqrt(n)    # t-distribution boundary
upper/sqrt(n)                                                             # scaled boundary
lower <- max(ds[which(pt(d_obs*sqrt(n),nu,ds*sqrt(n))>0.975)])*sqrt(n)
lower/sqrt(n)

Below is a situation for the case when the observed $d$ is 0.1 and the sample sizes are $n_1 = n_2 = 5$. In this case the confidence interval is

$$CI: -1.43619,1.337479$$

In the image you see how $d$ is distributed for different true values of $d$ (these distributions are scaled non-central t-distributions).

The red curve is the distribution of observed $d$ if the true value of $d$ would be equal to the upper limit of the confidence interval $1.337479$. In that case the observation of $d=0.1$ or lower would only occur in 2.5% of the cases (the red shaded area).

The blue curve is the distribution of the observed $d$ if the true value of $d$ would be equal to the lower limit of the confidence interval $-1.143619$. In that case the observation of $d=0.1$ or higher would only occur in 2.5% of the cases (the blue shaded area).

example for CI computation

$\endgroup$
16
  • $\begingroup$ @machine "I need to pool Cohens d and its SE with Rubins Rule after multiple imputation" If you have the original data (absolute effect size, estimated variance and sample size) instead of only cohen's d values, then I imagine it might be better to pool the original data and from that compute a pooled value for effect size $d$. $\endgroup$ Commented Nov 5, 2020 at 10:10
  • $\begingroup$ @machine This Rubins Rule is new to me, but what I understand is that you apply it to multiple estimates derived from the same data (for instance multiple estimates based on different ways of imputation), and then you have an estimate of the variance due to two sources (within and between) which you can add together... But why not estimate the absolute effect size instead of cohen's d (which is a scaled effect size) in this way, and then compute cohen's d based on this pooled absolute effect size and the estimate of it's variance? I will make a question about this. $\endgroup$ Commented Nov 5, 2020 at 10:27
  • $\begingroup$ Cohen's d is $\frac{\bar{x}_1-\bar{x}_2}{\hat{\sigma}}$ which is a standardized form of the absolute effect size ${\bar{x}_1-\bar{x}_2}$ $\endgroup$ Commented Nov 5, 2020 at 10:32
  • $\begingroup$ stats.stackexchange.com/questions/495174 $\endgroup$ Commented Nov 5, 2020 at 10:51
  • $\begingroup$ @machine What sort of data imputation are you doing? You have two groups and there are missing values. Why are you using multiple imputation? Is the comparison pairwise data? Then maybe use onlinelibrary.wiley.com/doi/abs/10.1002/bimj.201100053 or jstor.org/stable/2335098 (and there are many other methods) Is the comparison unpaired then why use imputation to replace the missing values? $\endgroup$ Commented Nov 5, 2020 at 12:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.