# Is there any measure of Effect Size for differences assessed with Dunnett method?

I know I could use the effect size pairwise, but I have a longitudinal data, where subjects were measures multiple times and there exists ICC > 0.5. I guess the pooled SD may be altered by this fact at each time point. Is there any dedicated measure, that takes also the intra-subject covariance into calculations, or can I just pretend the differences t1-t0, t2-t0 .... tn-t0 are independent and calculate, say, Cohen's D?

• The intra-subject covariance is relevant but only for the standard error of Cohen's d. For the computation of Cohen's d, you simply want the pooled standard deviation. In this case, you want a single pooled SD across all time points so that you are standardizing the mean differences using a consistent denominator. The standard error then needs to incorporate the ICC. I hope this makes sense. Jul 14, 2020 at 14:23
• Yes, thank you very much. Currently I only report the effect size alone, but I can imagine if my collaborators ask me to calculate also confidence intervals for it. I am not sure how to do this. There are procedures for paired data, but here it's multiple assessment procedure. Anyway, thank you! Jul 14, 2020 at 14:52
• I'm not sure if I get it. Let's assume I have 3 time points: t0, t1, t2. So I have 2 differences versus t0: t1-t0 and t2-t0. So should I calculate the effect size for each difference separately, but the pooled SD must be spanned across t0, t1 AND t2? Even if I do this pairwise? If the variances will differ, can i still do that? I think I understand - to refer all differences to the common variance (from all time points) rather than refer all differences only to those (each 2) pooled variance? Jul 14, 2020 at 15:58
• Is this what you meant? stackoverflow.com/a/57426868 (also includes the covariance) Jul 14, 2020 at 16:15
• I'm afraid it cannot be. The time points suggest dependent samples. And the formula for paired Cohen's d involves correlation between 2 time points. This suggests, that I have to run it pairwise. memory.psych.mun.ca/models/stats/effect_size.shtml Jul 14, 2020 at 16:24

This a longer and more complete answer than I provided in my comments. The logic of Cohen's $$d$$ is to standardized the mean difference by the natural variability on the dependent variable. Ideally, this would be the population standard deviation, $$\sigma$$, for the population of interest. We typically estimate this value from the sample data, using the pooled within-groups standard deviation. Why the within-groups and not the overall standard deviation? To the extent that the group contrast (i.e., treatment) has an effect, it will increase the variability in the sample. Thus, by using the pooled within-groups standard deviation, we are using the natural variability on this measure without any influence of the independent variable that defines the two groups. Thus, Cohen's $$d$$ is computed as: $$d = \frac{\overline{x}_1 - \overline{x}_2}{s_{pooled}}$$ where $$\overline{x}_1$$ is the mean for group 1, $$\overline{x}_2$$ is the mean for group 2, and $$s_{pooled}$$ is computed as $$s_{pooled} = \sqrt{\frac{s_1^2(n_1-1) + s_2^2(n_2-1)}{n_1 + n_2 - 2}}$$ where $$s$$ is the standard deviation and $$n$$ is the sample size for the respective groups, 1 and 2.

If you have three or more means and want to compute all pairwise Cohen's $$d$$ values, you want to use a common $$s_{pooled}$$. Otherwise, you are standardizing the mean difference using a different standard. You can think of the denominator as a "standard" ruler being used to measure the mean difference. Using a separate $$s_{pooled}$$, each based on the standard deviations for the two groups of interest, when computing all pairwise $$d$$s is like using a different ruler for each comparison. As a result, the $$d$$s are not directly comparable. The above formula for $$s_{pooled}$$ can easily be extended to any number of standard deviations, as shown below: $$s_{pooled} = \sqrt{\frac{\sum [s_j^2(n_j-1)]}{\sum (n_j - 1)}} ~.$$ Note that the equation above returns $$\sqrt{\text{MS}_{\text{residual}}}$$ from a one-way ANOVA.

The above assumes independent samples. For dependent samples, such as a repeated measures design, we can define Cohen's $$d$$ in the same way. (Note, however, that Cohen did not provide a version of $$d$$ for dependent samples.) If we want Cohen's $$d$$ to have the same interpretation, then the mean difference must still be standardized relative to the natural (i.e., unaffected by treatment) variability on the dependent variable. The above equations can be used. Because the time 1 and time 2 samples sizes should be equal, the pooled standard deviation can also be simplified to $$s_{pooled} = \sqrt{\frac{s_1^2 + s_2^2}{2}} .$$ With more than two time points, this can easily be generalized as it is just the square root of the mean of the variances (when the $$n$$s are equal) or of the weights mean when the $$n$$s are unequal, weighting by the degrees of freedom for each variance (i.e., $$n-1$$).

In the literature, you can find an alternative effect size for dependent or repeated samples that is standardized based on variability in change. This is technically not a Cohen's $$d$$, although we will continue to use $$d$$ in the notation below as that is what is common in the literature. This alternative effect size is defined as $$d = \frac{\overline{x}_2 - \overline{x}_1 }{s_\Delta} = \frac{\overline{\Delta}}{s_\Delta},$$ where $$s_\Delta$$ is the standard deviation of the time 1 and time 2 differences, that is, the standard deviation in change ($$\Delta$$). The interpretation of this $$d$$ is less straightforward relative to Cohen's $$d$$. It reflects the mean difference in standard deviation units of change. The problem with this index as a descriptive index is that a treatment that produces consistent change across individuals may have a larger effect size than a treatment that produces inconsistent change across individuals, even if the latter produced a larger mean difference. We can see this by examining the relationship between $$s_{pooled}$$ and $$s_\Delta$$, shown here: $$s_\Delta = s_{pooled} \sqrt{2(1-r)} ,$$ where $$r$$ is the correlation between the time 1 and time 2 scores. When $$r$$ equals 0.50, $$s_\Delta$$ and $$s_{pooled}$$ will be equal. For values of $$r$$ greater than 0.5, $$s_\Delta$$ will be smaller. For example, if $$r = 0.80$$, then $$s_{pooled}$$ will be multipled by 0.63, producing an $$s_\Delta$$ that is 37% smaller than $$s_{pooled}$$. As the variability in change decreases, $$d$$ increases for constant mean differences.

Thus, for descriptive purposes, Cohen's $$d$$ should be calculated the same whether the data reflect independent samples or dependent samples. However, there is a critical difference in how the standard error is computed. For the independent samples, the standard error for Cohen's $$d$$ is computed as: $$s_d = \sqrt{\left( \frac{n_1 + n_2}{n_1n_2}\right)\frac{d^2}{2(n_1+n_2}} ~.$$ Note that this is technically an approximation. See: [https://stats.stackexchange.com/questions/144084/variance-of-cohens-d-statistic].1

For Cohen's $$d$$ based on dependent samples, the variance is: $$s_{d_\Delta} = \sqrt{\left(\frac{2(1-r)}{n}\right)\frac{d_\Delta^2}{2n}} ~.$$

A final side note. The above has focused on Cohen's $$d$$ for descriptive purposes, such as reporting on the results of a study or for use in a meta-analysis. Power analysis programs often compute $$d$$ using the standard deviation or the residuals of the dependent variable in the denominator, thus taking into account the power advantage of any covariates in the model. Similarly, in a dependent samples case, the denominator would include information on the pre-post correlation, thus incorporating any power gain (if $$r > .5$$) or power loss (if $$r < .5$$) into the power calculations. This is a simple way to estimate power but produces a $$d$$ that lacks any natural interpretation.