1
$\begingroup$

I want to calculate cohen's d with confidence intervals for a paired samples designs.

Some authors suggest that you use the paired t test value to adjust for the correlation between measures (Rosenthal, 1991).

However, Dunlop et al 1996 suggest that the correlation between the paired samples should not be included. In particular, if such a correlation is used, then results are not readily comparable to between subjects effects. Instead, they recommend using independent samples formula:

$$d= \frac{\mu_1-\mu_2}{\sigma_\textrm{pooled}}$$

Question

How do you estimate confidence intervals for Cohen's d in a paired samples design where Cohen's d uses the formula above?

The way I have estimated it so far is to use the R function ci.smd from the MBESS package. This function is designed for independent samples tests and takes the sizes of two independent groups as input.

In the code below I use i to represent the number of subjects (equal, as the same participants are in each group) and use an effect size of 0.8 as an example.

library(MBESS)
x.ci <- ci.smd(smd=0.8, n.1=i,n.2=i)

Is it appropriate to use such function ci.smd for confidence intervals for paired samples?

$\endgroup$

3 Answers 3

1
$\begingroup$

This is just about Dunlop et al because I've got no time to go through the code.

They have a point when the design of the study could be between or within. Nevertheless, I think both effect sizes could easily be reported. Minimally, if you follow their advice, you should also report your correlation between your groups so that the repeated measures effect size can be calculated.

But their argument often gets extended to cases where you can only do the experiment within subject. I remember some property of differences between people's ears that gets studied where the literature strongly endorsed Dunlop. In that case, there's no way to do the study as an independent groups design and therefore no concern about comparing the effect sizes observed across repeated measures and independent groups.

It depends on the kind of study you're doing whether you follow Dunlop et al. Are there both independent and repeated measures designs possible? Is there a related literature you want to consider that can have independent measures? If so, include an independent groups effect size.

And also make sure you publish how you pool your effect variance. There are a variety of ways it's done and it's important to specify.

$\endgroup$
2
  • $\begingroup$ Thanks for the reply. To give a specific example. I am aiming to compute effect sizes from published data, which is a repeated measures design. Suppose the data only reports M and SD for two conditions, to which all subjects are exposed: Con 1(M=651,SD=87), Con 2 (M=611,SD=80) It seems that (according to Dunlop) the best/only way to compute d is M1-M2/pooled sd. (pooled sd is the square root of the average of the squared deviations- as used in calculators such as this: uccs.edu/lbecker/index.html) How are confidence intervals constructed for a d computed in this way? $\endgroup$
    – user21879
    Mar 12, 2013 at 13:04
  • $\begingroup$ You would need to have an estimate of the correlation between the two conditions in order to calculate the confidence interval even if that correlation was not used in the formula. This is because the standard error of M1 - M2 is influenced by this correlation. $\endgroup$ May 12, 2013 at 8:01
1
$\begingroup$

Using Dunlop et al

Ultimately, dividing the difference by a number is just a means of putting the difference between means on a more meaningful metric. I generally don't find the correlation between paired samples to be relevant to defining a meaningful metric, but there might be cases where it is meaningful. In particular, where comparisons of effects are being made between within and between subjects designs, it is particularly important to use a common definition of Cohen's d. I've read meta-analyses where researchers have failed to make this distinction and as a consequence have falsely concluded that within-subject designs yielded larger effect sizes!

Confidence intervals for paired samples d

Confidence interval formulas designed for independent groups will give inaccurate answers for paired samples designs. The correlation between paired samples will generally reduce the standard error of the difference between means. Thus, confidence intervals will be smaller.

I'm not sure of what the exact formula is for a between subjects d on a paired-samples design. But here are a couple of thoughts:

  • You should be able to get an approximate idea of your confidence intervals by getting the confidence intervals from your raw data from a paired samples t-test and calculating the d that corresponds to the lower and upper 95% raw difference using the pooled SD. However, this wont incorporate uncertainty over the standard deviation.
  • You could calculate bootstrapped confidence intervals. Just calculate the d of interest for each bootstrapped sample and then extract the relevant quantiles (e.g., .025 and .0975 for 95%CI).
$\endgroup$
0
$\begingroup$

This is an old post, but a recent paper went through the computation of Cohen's $d$ (unbiased as well as the biased version) for both within and between-subject design. It also offers a R function that does all the computation provided that the data are available.

See https://doi.org/10.20982/tqmp.14.4.p242

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.