55
$\begingroup$

Imagine the following common design:

  • 100 participants are randomly allocated to either a treatment or a control group
  • the dependent variable is numeric and measured pre- and post- treatment

Three obvious options for analysing such data are:

  • Test the group by time interaction effect in mixed ANOVA
  • Do an ANCOVA with condition as the IV and the pre- measure as the covariate and post measure as the DV
  • Do a t-test with condition as the IV and pre-post change scores as the DV

Question:

  • What is the best way to analyse such data?
  • Are there reasons to prefer one approach over another?
| cite | improve this question | |
$\endgroup$
37
$\begingroup$

There is a huge literature around this topic (change/gain scores), and I think the best references come from the biomedical domain, e.g.

Senn, S (2007). Statistical issues in drug development. Wiley (chap. 7 pp. 96-112)

In biomedical research, interesting work has also been done in the study of cross-over trials (esp. in relation to carry-over effects, although I don't know how applicable it is to your study).

From Gain Score t to ANCOVA F (and vice versa), from Knapp & Schaffer, provides an interesting review of ANCOVA vs. t approach (the so-called Lord's Paradox). The simple analysis of change scores is not the recommended way for pre/post design according to Senn in his article Change from baseline and analysis of covariance revisited (Stat. Med. 2006 25(24)). Moreover, using a mixed-effects model (e.g. to account for the correlation between the two time points) is not better because you really need to use the "pre" measurement as a covariate to increase precision (through adjustment). Very briefly:

  • The use of change scores (post $-$ pre, or outcome $-$ baseline) does not solve the problem of imbalance; the correlation between pre and post measurement is < 1, and the correlation between pre and (post $-$ pre) is generally negative -- it follows that if the treatment (your group allocation) as measured by raw scores happens to be an unfair disadvantage compared to control, it will have an unfair advantage with change scores.
  • The variance of the estimator used in ANCOVA is generally lower than that for raw or change scores (unless correlation between pre and post equals 1).
  • If the pre/post relationships differ between the two groups (slope), it is not as much of a problem than for any other methods (the change scores approach also assumes that the relationship is identical between the two groups -- the parallel slope hypothesis).
  • Under the null hypothesis of equality of treatment (on the outcome), no interaction treatment x baseline is expected; it is dangerous to fit such a model, but in this case one must use centered baselines (otherwise, the treatment effect is estimated at the covariate origin).

I also like Ten Difference Score Myths from Edwards, although it focuses on difference scores in a different context; but here is an annotated bibliography on the analysis of pre-post change (unfortunately, it doesn't cover very recent work). Van Breukelen also compared ANOVA vs. ANCOVA in randomized and non-randomized setting, and his conclusions support the idea that ANCOVA is to be preferred, at least in randomized studies (which prevent from regression to the mean effect).

| cite | improve this answer | |
$\endgroup$
  • 1
    $\begingroup$ Just to clarify: do you mean that the ANCOVA with pre-test scores as covariates is the best option? $\endgroup$ – mkt - Reinstate Monica Sep 30 '19 at 14:09
  • $\begingroup$ @chl: The link to the annotated bibliography is broken. $\endgroup$ – kjetil b halvorsen May 2 at 16:22
  • $\begingroup$ Thanks @kjetilbhalvorsen. This should be fixed now! $\endgroup$ – chl Sep 14 at 18:03
  • 1
    $\begingroup$ @mkt-ReinstateMonica This is my understanding. You may also refer to arguments raised by Frank Harrell in his main book, Regression Modeling Strategies, or his BBR textbook (see also Statistical Errors in the Medical Literature), or Stephen Seen's article on the analysis of change from baseline. $\endgroup$ – chl Sep 14 at 18:09
17
$\begingroup$

Daniel B. Wright discusses this in section 5 of his article Making Friends with your Data. He suggests (p.130):

The only procedure that is always correct in this situation is a scatterplot comparing the scores at time 2 with those at time 1 for the different groups. In most cases you should analyse the data in several ways. If the approaches give different results ... think more carefully about the model implied by each.

He recommends the following articles as further reading:

  • Hand, D. J. (1994). Deconstructing statistical questions. Journal of the Royal Statistical Society: A, 157, 317–356.
  • Lord, F. M. (1967). A paradox in the interpretation of group comparisons. Psychological Bulletin, 72, 304–305. Free PDF
  • Wainer, H. (1991). Adjusting for differential base rates: Lord’s paradox again. Psychological Bulletin, 109, 147–151. Free PDF
| cite | improve this answer | |
$\endgroup$
9
$\begingroup$

The most common strategies would be:

  1. Repeated measures ANOVA with one within-subject factor (pre vs. post-test) and one between-subject factor (treatment vs. control).
  2. ANCOVA on the post-treatment scores, with pre-treatment score as a covariate and treatment as an independent variable. Intuitively, the idea is that a test of the differences between both groups is really what you are after and including pre-test scores as a covariate can increase power compared to a simple t-test or ANOVA.

There are many discussions on the interpretation, assumptions, and apparently paradoxical differences between these two approaches and on more sophisticated alternatives (especially when participants cannot be randomly assigned to treatment) but they remain pretty standard, I think.

One important source of confusion is that for the ANOVA, the effect of interest is most likely the interaction between time and treatment and not the treatment main effect. Incidentally, the F-test for this interaction term will yield exactly the same result than an independent sample t-test on gain scores (i.e. scores obtained by subtracting the pre-test score from the post-test score for each participant) so you might also go for that.

If all this is too much, you don't have time to figure it out, and cannot obtain some help from a statistician, a quick and dirty but by no means entirely absurd approach would be to simply compare the post-test scores with an independent sample t-test, ignoring pre-test values. This only makes sense if participants were in fact randomly assigned to the treatment or control group.

Finally, that's not in itself a very good reason to choose it but I suspect approach 2 above (ANCOVA) is what currently passes for the right approach in psychology so if you choose something else you might have to explain the technique in detail or to justify yourself to someone who is convinced, e.g. that “gain scores are known to be bad”.

| cite | improve this answer | |
$\endgroup$
  • 2
    $\begingroup$ I'd say the first recommendation, repeated measures ANOVA, is not appropriate for analyzing pre-post data. Is treatment coded to 0 in the intervention group at baseline? Either way, this reintroduces the Hawthorne effect. Systematic differences in pre/post among the controls is chocked up to random variation. The RM ANCOVA is justified when there are multiple measurements during a post-period, and baseline values are still adjusted as a covariate or used as a gain-score. $\endgroup$ – AdamO Dec 19 '17 at 18:49
2
$\begingroup$

ANCOVA and repeated measures/mixed model for interaction term are testing two different hypothesis. Refer to this article: ariticle 1 and this article: article 2

| cite | improve this answer | |
$\endgroup$
-2
$\begingroup$

Since you have two means (either of a specific item, or of the sum of the inventory), there's no reason to consider an ANOVA. A paired t-test is probably appropriate; this may help you choose which t-test you need.

Do you want to look at item-specific results, or at overall scores? If you want to do an item analysis, this might be a useful starting place.

| cite | improve this answer | |
$\endgroup$
  • 4
    $\begingroup$ What about the control group? A paired t-test on all the data sounds like a bad idea and certainly does not address the main question (is the treatment effective?). A paired t-test restricted to the treatment group is a plausible strategy but ignoring the control group throws away a lot of data and makes for much weaker evidence that the intervention is in fact the active ingredient. ANOVA is in fact a common - if often criticized - way to analyze this design. $\endgroup$ – Gala May 1 '13 at 16:23

Not the answer you're looking for? Browse other questions tagged or ask your own question.