3
$\begingroup$

I am interesting in learning if there is difference between the control and the treatment group in a paper I am reading. Unfortunately the paper only compares the respective conditions to a 0 effect size. The paper does not have open data or report standard deviations.

The relevent results reported in the paper are:

A one-sample t test on the recommended IAT measure indicated that the mean D score for the participants in the control condition (M = −.19 was significantly less than zero, t(29) = −2.50, p = .019. The estimated Cohen’s d for this effect was .46. In contrast, a one-sample t test indicated that the mean D score for the participants in the perspective taking condition (M = −.13) was not significantly less than zero, t(29) = −1.61, p = .119.

Is it possible to compare the two effect sizes to learn about the difference between the two conditions instead?

$\endgroup$
3
$\begingroup$

I assume that what you are interested in is the difference in differences. Simply differencing the mean D for the treatment and control is this value (i.e., -.19 - (-.13) = -.06). You can compute a t-test on this difference by recovering the standard deviations of the difference scores from the t-value: $$ s_D = \sqrt{\frac{M_D^2 N}{t^2}} . $$ For the control group, this is: $$ s_D = \sqrt{\frac{-.19^2(30)}{-2.50^2}} = 0.416 . $$ For the treatment group this is: $$ s_D = \sqrt{\frac{-.13^2(30)}{-1.61^2}} = 0.442. $$ With these values, you can compute a t-test comparing these two mean difference scores: $$ t = \frac{M_{D_{Tx}} - M_{D_{Cg}}}{ \sqrt{ \left(\frac{ s_1^2(N_1 - 1) + s_2^2(N_2 - 1)}{N_1 + N_2 - 2}\right) \left(\frac{N_1 + N_2}{N_1N_2}\right) } } , $$ $$ t = \frac{-0.19 - (-0.13)}{ \sqrt{ \left(\frac{ 0.442^2(30 - 1) + 0.416(30 - 1)}{30 + 30 - 2}\right) \left(\frac{30 + 30}{30(30)}\right) } } = -0.1397 . $$

This is a more meaningful test of any treatment effect and is the interaction in a 2 by 2 split-plot ANOVA (time by treatment).

Note that the authors computed Cohen's d as the mean difference divided by the standard deviation of the mean difference. A more standard Cohen's d for paired data is to divide by either the pre-test standard deviation or both the pooled pre and post-test standard deviations. Dividing by the standard deviation in differences is standardizing change relative to variability in change across people rather than simply variability in the measure across people. I think that the latter is far more interpretable.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.