Best practice when analysing pre-post treatment-control designs

Question

Imagine the following common design:

• 100 participants are randomly allocated to either a treatment or a control group
• the dependent variable is numeric and measured pre- and post- treatment

Three obvious options for analysing such data are:

• Test the group by time interaction effect in mixed ANOVA
• Do an ANCOVA with condition as the IV and the pre- measure as the covariate and post measure as the DV
• Do a t-test with condition as the IV and pre-post change scores as the DV

Question:

• What is the best way to analyse such data?
• Are there reasons to prefer one approach over another?

Answer 1

There is a huge literature around this topic (change/gain scores), and I think the best references come from the biomedical domain, e.g.

Senn, S (2007). Statistical issues in drug development. Wiley (chap. 7 pp. 96-112)

In biomedical research, interesting work has also been done in the study of cross-over trials (esp. in relation to carry-over effects, although I don't know how applicable it is to your study).

From Gain Score t to ANCOVA F (and vice versa), from Knapp & Schaffer, provides an interesting review of ANCOVA vs. t approach (the so-called Lord's Paradox). The simple analysis of change scores is not the recommended way for pre/post design according to Senn in his article Change from baseline and analysis of covariance revisited (Stat. Med. 2006 25(24)). Moreover, using a mixed-effects model (e.g. to account for the correlation between the two time points) is not better because you really need to use the "pre" measurement as a covariate to increase precision (through adjustment). Very briefly:

• The use of change scores (post $-$ pre, or outcome $-$ baseline) does not solve the problem of imbalance; the correlation between pre and post measurement is < 1, and the correlation between pre and (post $-$ pre) is generally negative -- it follows that if the treatment (your group allocation) as measured by raw scores happens to be an unfair disadvantage compared to control, it will have an unfair advantage with change scores.
• The variance of the estimator used in ANCOVA is generally lower than that for raw or change scores (unless correlation between pre and post equals 1).
• If the pre/post relationships differ between the two groups (slope), it is not as much of a problem than for any other methods (the change scores approach also assumes that the relationship is identical between the two groups -- the parallel slope hypothesis).
• Under the null hypothesis of equality of treatment (on the outcome), no interaction treatment x baseline is expected; it is dangerous to fit such a model, but in this case one must use centered baselines (otherwise, the treatment effect is estimated at the covariate origin).

I also like Ten Difference Score Myths from Edwards, although it focuses on difference scores in a different context; but here is an annotated bibliography on the analysis of pre-post change (unfortunately, it doesn't cover very recent work). Van Breukelen also compared ANOVA vs. ANCOVA in randomized and non-randomized setting, and his conclusions support the idea that ANCOVA is to be preferred, at least in randomized studies (which prevent from regression to the mean effect).

Answer 2

Daniel B. Wright discusses this in section 5 of his article Making Friends with your Data. He suggests (p.130):

The only procedure that is always correct in this situation is a scatterplot comparing the scores at time 2 with those at time 1 for the different groups. In most cases you should analyse the data in several ways. If the approaches give different results ... think more carefully about the model implied by each.

He recommends the following articles as further reading:

• Hand, D. J. (1994). Deconstructing statistical questions. Journal of the Royal Statistical Society: A, 157, 317–356.
• Lord, F. M. (1967). A paradox in the interpretation of group comparisons. Psychological Bulletin, 72, 304–305. Free PDF
• Wainer, H. (1991). Adjusting for differential base rates: Lord’s paradox again. Psychological Bulletin, 109, 147–151. Free PDF

Answer 3

The most common strategies would be:

1. Repeated measures ANOVA with one within-subject factor (pre vs. post-test) and one between-subject factor (treatment vs. control).
2. ANCOVA on the post-treatment scores, with pre-treatment score as a covariate and treatment as an independent variable. Intuitively, the idea is that a test of the differences between both groups is really what you are after and including pre-test scores as a covariate can increase power compared to a simple t-test or ANOVA.

There are many discussions on the interpretation, assumptions, and apparently paradoxical differences between these two approaches and on more sophisticated alternatives (especially when participants cannot be randomly assigned to treatment) but they remain pretty standard, I think.

One important source of confusion is that for the ANOVA, the effect of interest is most likely the interaction between time and treatment and not the treatment main effect. Incidentally, the F-test for this interaction term will yield exactly the same result than an independent sample t-test on gain scores (i.e. scores obtained by subtracting the pre-test score from the post-test score for each participant) so you might also go for that.

If all this is too much, you don't have time to figure it out, and cannot obtain some help from a statistician, a quick and dirty but by no means entirely absurd approach would be to simply compare the post-test scores with an independent sample t-test, ignoring pre-test values. This only makes sense if participants were in fact randomly assigned to the treatment or control group.

Finally, that's not in itself a very good reason to choose it but I suspect approach 2 above (ANCOVA) is what currently passes for the right approach in psychology so if you choose something else you might have to explain the technique in detail or to justify yourself to someone who is convinced, e.g. that “gain scores are known to be bad”.

Answer 4

ANCOVA and repeated measures/mixed model for interaction term are testing two different hypothesis. Refer to this article: ariticle 1 and this article: article 2

Answer 5

Since you have two means (either of a specific item, or of the sum of the inventory), there's no reason to consider an ANOVA. A paired t-test is probably appropriate; this may help you choose which t-test you need.

Do you want to look at item-specific results, or at overall scores? If you want to do an item analysis, this might be a useful starting place.