How to estimate Cohen's D using the independent groups definition but on a paired samples data?

Question

I want to calculate cohen's d with confidence intervals for a paired samples designs.

Some authors suggest that you use the paired t test value to adjust for the correlation between measures (Rosenthal, 1991).

However, Dunlop et al 1996 suggest that the correlation between the paired samples should not be included. In particular, if such a correlation is used, then results are not readily comparable to between subjects effects. Instead, they recommend using independent samples formula:

$$d= \frac{\mu_1-\mu_2}{\sigma_\textrm{pooled}}$$

Question

How do you estimate confidence intervals for Cohen's d in a paired samples design where Cohen's d uses the formula above?

The way I have estimated it so far is to use the R function ci.smd from the MBESS package. This function is designed for independent samples tests and takes the sizes of two independent groups as input.

In the code below I use i to represent the number of subjects (equal, as the same participants are in each group) and use an effect size of 0.8 as an example.

library(MBESS)
x.ci <- ci.smd(smd=0.8, n.1=i,n.2=i)

Is it appropriate to use such function ci.smd for confidence intervals for paired samples?

Answer 1

This is just about Dunlop et al because I've got no time to go through the code.

They have a point when the design of the study could be between or within. Nevertheless, I think both effect sizes could easily be reported. Minimally, if you follow their advice, you should also report your correlation between your groups so that the repeated measures effect size can be calculated.

But their argument often gets extended to cases where you can only do the experiment within subject. I remember some property of differences between people's ears that gets studied where the literature strongly endorsed Dunlop. In that case, there's no way to do the study as an independent groups design and therefore no concern about comparing the effect sizes observed across repeated measures and independent groups.

It depends on the kind of study you're doing whether you follow Dunlop et al. Are there both independent and repeated measures designs possible? Is there a related literature you want to consider that can have independent measures? If so, include an independent groups effect size.

And also make sure you publish how you pool your effect variance. There are a variety of ways it's done and it's important to specify.

Answer 2

Using Dunlop et al

Ultimately, dividing the difference by a number is just a means of putting the difference between means on a more meaningful metric. I generally don't find the correlation between paired samples to be relevant to defining a meaningful metric, but there might be cases where it is meaningful. In particular, where comparisons of effects are being made between within and between subjects designs, it is particularly important to use a common definition of Cohen's d. I've read meta-analyses where researchers have failed to make this distinction and as a consequence have falsely concluded that within-subject designs yielded larger effect sizes!

Confidence intervals for paired samples d

Confidence interval formulas designed for independent groups will give inaccurate answers for paired samples designs. The correlation between paired samples will generally reduce the standard error of the difference between means. Thus, confidence intervals will be smaller.

I'm not sure of what the exact formula is for a between subjects d on a paired-samples design. But here are a couple of thoughts:

• You should be able to get an approximate idea of your confidence intervals by getting the confidence intervals from your raw data from a paired samples t-test and calculating the d that corresponds to the lower and upper 95% raw difference using the pooled SD. However, this wont incorporate uncertainty over the standard deviation.
• You could calculate bootstrapped confidence intervals. Just calculate the d of interest for each bootstrapped sample and then extract the relevant quantiles (e.g., .025 and .0975 for 95%CI).

Answer 3

This is an old post, but a recent paper went through the computation of Cohen's $d$ (unbiased as well as the biased version) for both within and between-subject design. It also offers a R function that does all the computation provided that the data are available.

See https://doi.org/10.20982/tqmp.14.4.p242