Is there any measure of Effect Size for differences assessed with Dunnett method?

Question

I know I could use the effect size pairwise, but I have a longitudinal data, where subjects were measures multiple times and there exists ICC > 0.5. I guess the pooled SD may be altered by this fact at each time point. Is there any dedicated measure, that takes also the intra-subject covariance into calculations, or can I just pretend the differences t1-t0, t2-t0 .... tn-t0 are independent and calculate, say, Cohen's D?

Answer 1

This a longer and more complete answer than I provided in my comments. The logic of Cohen's $d$ is to standardized the mean difference by the natural variability on the dependent variable. Ideally, this would be the population standard deviation, $\sigma$, for the population of interest. We typically estimate this value from the sample data, using the pooled within-groups standard deviation. Why the within-groups and not the overall standard deviation? To the extent that the group contrast (i.e., treatment) has an effect, it will increase the variability in the sample. Thus, by using the pooled within-groups standard deviation, we are using the natural variability on this measure without any influence of the independent variable that defines the two groups. Thus, Cohen's $d$ is computed as: $$ d = \frac{\overline{x}_1 - \overline{x}_2}{s_{pooled}} $$ where $\overline{x}_1$ is the mean for group 1, $\overline{x}_2$ is the mean for group 2, and $s_{pooled}$ is computed as $$ s_{pooled} = \sqrt{\frac{s_1^2(n_1-1) + s_2^2(n_2-1)}{n_1 + n_2 - 2}} $$ where $s$ is the standard deviation and $n$ is the sample size for the respective groups, 1 and 2.

If you have three or more means and want to compute all pairwise Cohen's $d$ values, you want to use a common $s_{pooled}$. Otherwise, you are standardizing the mean difference using a different standard. You can think of the denominator as a "standard" ruler being used to measure the mean difference. Using a separate $s_{pooled}$, each based on the standard deviations for the two groups of interest, when computing all pairwise $d$s is like using a different ruler for each comparison. As a result, the $d$s are not directly comparable. The above formula for $s_{pooled}$ can easily be extended to any number of standard deviations, as shown below: $$ s_{pooled} = \sqrt{\frac{\sum [s_j^2(n_j-1)]}{\sum (n_j - 1)}} ~. $$ Note that the equation above returns $\sqrt{\text{MS}_{\text{residual}}}$ from a one-way ANOVA.

The above assumes independent samples. For dependent samples, such as a repeated measures design, we can define Cohen's $d$ in the same way. (Note, however, that Cohen did not provide a version of $d$ for dependent samples.) If we want Cohen's $d$ to have the same interpretation, then the mean difference must still be standardized relative to the natural (i.e., unaffected by treatment) variability on the dependent variable. The above equations can be used. Because the time 1 and time 2 samples sizes should be equal, the pooled standard deviation can also be simplified to $$ s_{pooled} = \sqrt{\frac{s_1^2 + s_2^2}{2}} . $$ With more than two time points, this can easily be generalized as it is just the square root of the mean of the variances (when the $n$s are equal) or of the weights mean when the $n$s are unequal, weighting by the degrees of freedom for each variance (i.e., $n-1$).

In the literature, you can find an alternative effect size for dependent or repeated samples that is standardized based on variability in change. This is technically not a Cohen's $d$, although we will continue to use $d$ in the notation below as that is what is common in the literature. This alternative effect size is defined as $$ d = \frac{\overline{x}_2 - \overline{x}_1 }{s_\Delta} = \frac{\overline{\Delta}}{s_\Delta}, $$ where $s_\Delta$ is the standard deviation of the time 1 and time 2 differences, that is, the standard deviation in change ($\Delta$). The interpretation of this $d$ is less straightforward relative to Cohen's $d$. It reflects the mean difference in standard deviation units of change. The problem with this index as a descriptive index is that a treatment that produces consistent change across individuals may have a larger effect size than a treatment that produces inconsistent change across individuals, even if the latter produced a larger mean difference. We can see this by examining the relationship between $s_{pooled}$ and $s_\Delta$, shown here: $$ s_\Delta = s_{pooled} \sqrt{2(1-r)} , $$ where $r$ is the correlation between the time 1 and time 2 scores. When $r$ equals 0.50, $s_\Delta$ and $s_{pooled}$ will be equal. For values of $r$ greater than 0.5, $s_\Delta$ will be smaller. For example, if $r = 0.80$, then $s_{pooled}$ will be multipled by 0.63, producing an $s_\Delta$ that is 37% smaller than $s_{pooled}$. As the variability in change decreases, $d$ increases for constant mean differences.

Thus, for descriptive purposes, Cohen's $d$ should be calculated the same whether the data reflect independent samples or dependent samples. However, there is a critical difference in how the standard error is computed. For the independent samples, the standard error for Cohen's $d$ is computed as: $$ s_d = \sqrt{\left( \frac{n_1 + n_2}{n_1n_2}\right)\frac{d^2}{2(n_1+n_2}} ~. $$ Note that this is technically an approximation. See: [https://stats.stackexchange.com/questions/144084/variance-of-cohens-d-statistic].1

For Cohen's $d$ based on dependent samples, the variance is: $$ s_{d_\Delta} = \sqrt{\left(\frac{2(1-r)}{n}\right)\frac{d_\Delta^2}{2n}} ~. $$

A final side note. The above has focused on Cohen's $d$ for descriptive purposes, such as reporting on the results of a study or for use in a meta-analysis. Power analysis programs often compute $d$ using the standard deviation or the residuals of the dependent variable in the denominator, thus taking into account the power advantage of any covariates in the model. Similarly, in a dependent samples case, the denominator would include information on the pre-post correlation, thus incorporating any power gain (if $r > .5$) or power loss (if $r < .5$) into the power calculations. This is a simple way to estimate power but produces a $d$ that lacks any natural interpretation.