Cohen's d between groups over time question

I have a problem trying to properly phrase my results.

I utilized:

$$D =\frac{(\text{M}_{2e} –\text{M}_{2c}) – (\text{M}_{1e}-\text{M}_{1c})}{\text{SD}_{1\text{pooled}}}$$

for pooled SD I used:

$$\text{SD}_{1\text{pooled}} = \sqrt{\left((n_1-1)s_1^2 + (n_2-1)s_2^2\right)\big /(n_1+n_2-2)}$$

My results for my manipulation check was $d = .08$. My results for my hypothesis testing was $d = .38$.

Can someone please explain what this means? I know that Cohen's $d$ of .08 is below the .2 criteria of small effect, and the Cohen's $d$ of .38 is a small to moderate effect. Where I am confused is what this means for a between groups over time comparison... Thank you or any insight you may be able to provide me.

-
Cohen's d does not have a universal scale. For experiments in psychology a d of 0.25 may be moderate, but in pharmacology it would be considered to be miniscule. –  Michael Lew Jan 16 '13 at 23:39
Thank you for that bit of info. Any help with how to word the result, specific to the measure being 'between groups over time' (i.e. does this mean the manipulation check didn't work, does it mean the hypothesis test has merit)? –  Robert Price Jan 17 '13 at 0:26
I am guessing that 1 and 2 are the time points e denotes the ‘experimental’ group and c a ‘control’ group, is that correct? –  Gala Jan 17 '13 at 11:31

It means that the change in the observed difference between the groups on your manipulation check measure is smaller than the change in the observed difference on the main outcome variable, relative to (an estimate of) their respective within-group variances. At least three things can contribute to produce such a result:

• The effect is simply smaller, in absolute terms, as reflected by the numerator (intuitively, it is easier to grasp what this means if you imagine that the two variables are measured with the same unit and a similar within-group variance). In your case, the effect is the change in differences so a small effect means that the between-group differences are of the same magnitude at both time points, i.e. that the experimental group was still as similar to the control group at time point 2 as it was at time point 1. Yet another way to put it is that the difference between group e and group c is the same at both time points. Conversely, a big effect would mean that group e and group c diverged from each other (positive effect) or converged (negative effect).
• One measure is less reliable than the other, thus inflating the denominator.
• Individuals differ more in one variable than the other, also inflating within-group variance in the denominator.

Also not to forget: The means in the numerator are estimates and if you estimate the standard deviation from the data, it would also contribute sampling variability to your estimate of $d$. Furthermore, the difference between means has a higher variability than the means themselves. With small samples, a difference between $d = .38$ and $d = .08$ might therefore not mean all that much. Note that this can be the case, even if there is a significant between-group difference on one of the variable but not on the other (i.e. “the difference between significant and non-significant is not necessarily itself significant”).

Another way to find out what is going on is to look at unstandardized effect sizes (i.e. the raw means or differences in means on the original scale, response times, etc.) Are they smaller than those reported in the literature? How big are they compared to the range of possible or “reasonable” values on this scale? etc.

Once you have done that, if it turns out that the between-group differences at both time points on the manipulation check measure are the same and you have confidence in your manipulation check, it will be very difficult to argue that the manipulation really is the “active ingredient” in your study. The hypothesis test might “have merit” statistically speaking but if you did not succeed in manipulating what you wanted to manipulate any other outcome must result from some other differences between the groups (improper randomization, lack of blinding or other defects in the procedure, etc.)

However, you did not specify if time point 1 is a measure before the manipulation (before-after design) or shortly after. In the latter case, it is also possible that the manipulation was effective immediately and that this effect is simply stable over time. You could compute an effect size for the between-group difference at each time point (standardized or unstandardized) to explore this possibility.

-