Is it possible to compute some measure of effect size from four means from a split-plot (both within- and between-subjects) 2x2 design?

Question

For a meta-analysis, we are interested in the comparison of two groups. From the articles, we can extract four means: two baseline measures on one of our dependent variables, and two measures collected after some manipulation took place. We also have the standard deviations for these means. The sample sizes are generally very small (which is one of the reasons why we usually have both pre- and post-measures; the researchers were probably aware of their low sample sizes and wanted to increase their power).

Although these means and SD's (variances) are often reported, the only statistics we have are usually from multivariate analyses, rendering them, at best, tricky to use (because the models (included covariates) differ substantially between studies). So we'd prefer to stick to the means and variances.

So, we want to compute the effect size. The go-to measure would be Cohen's d, to see how strongly these groups differ. We could ignore the baseline measure and compute d for post-test only, but this would of course yield a much lower power; ideally we'd look at change from baseline.

However, we don't have individual scores - we can't compute change over time per participant. We only have 'within-group variance' (to use oneway anova vocabulary).

I'd say this means that we might as well not have the baseline means: there is not way to use them to compute an effect size measure. But I want to make sure this is true - I'm not sure there's not something I miss. It feels like I do, but I'm not sure.

To provide an example, we have from one of the studies the following means (SD's):

               pre            post
control        33.2 (12.4)    33.1 (11.2)
experimental   29.5 (13)      30.9 (14.1)

Our goal is to express this association (columns 'within', rows 'between' subjects) in an effect size measure.

Is my intuition/fear, that we're forced to discard the first column because we can't remove or model the dependency between the two columns, correct? Or am I missing something and is there a way to use all available data after all?

Answer 1

This was discussed by Becker (1988) (and others). The suggested approach is to calculate the standardized mean change within each group and take the difference. The computation of the standardized mean change for one group is explained in this answer. So, compute this for the first and second group and denote those two values as $d_C$ and $d_E$ (and the corresponding sampling variances as $v_C$ and $v_E$). Then the difference in the standardized mean change is simply $$d = d_E - d_C$$ with sampling variance $$v = v_E + v_C.$$

One difficulty here is that the computation of $v_C$ and $v_E$ requires knowing the correlation between pre- and post-test scores within the groups (which is often not reported). So, you may have to make an educated guess about the correlation and then do sensitivity analyses at the end.