Take the 2-minute tour ×
Cross Validated is a question and answer site for people interested in statistics, machine learning, data analysis, data mining, and data visualization. It's 100% free, no registration required.

Imagine the following common design:

  • 100 participants are randomly allocated to either a treatment or a control group
  • the dependent variable is numeric and measured pre- and post- treatment

Three obvious options for analysing such data are:

  • Test the group by time interaction effect in mixed ANOVA
  • Do an ANCOVA with condition as the IV and the pre- measure as the covariate and post measure as the DV
  • Do a t-test with condition as the IV and pre-post change scores as the DV

Question:

  • What is the best way to analyse such data?
  • Are there reasons to prefer one approach over another?
share|improve this question
    
When you say "condition", do you mean group assignment? –  pmgjones Oct 10 '10 at 13:49
1  
@propofol: yes. apologies if my language is not clear. –  Jeromy Anglim Oct 11 '10 at 0:47
    
There also are parametric "N-of-1" methods for statistically evaluating temporal data for single observations. Example Application: ncbi.nlm.nih.gov/pubmed/2039432 Comparative Methods: europepmc.org/abstract/MED/10557859/… –  user31256 Oct 9 '13 at 6:53
3  

2 Answers 2

up vote 19 down vote accepted

There is a huge literature around this topic (change/gain scores), and I think the best references come from the biomedical domain, e.g.

Senn, S (2007). Statistical issues in drug development. Wiley (chap. 7 pp. 96-112)

In biomedical research, interesting work has also been done in the study of cross-over trials (esp. in relation to carry-over effects, although I don't know how applicable it is to your study).

From Gain Score t to ANCOVA F (and vice versa), from Knapp & Schaffer, provides an interesting review of ANCOVA vs. t approach (the so-called Lord's Paradox). The simple analysis of change scores is not the recommended way for pre/post design according to Senn in his article Change from baseline and analysis of covariance revisited (Stat. Med. 2006 25(24)). Moreover, using a mixed-effects model (e.g. to account for the correlation between the two time points) is not better because you really need to use the "pre" measurement as a covariate to increase precision (through adjustment). Very briefly:

  • The use of change scores (post $-$ pre, or outcome $-$ baseline) does not solve the problem of imbalance; the correlation between pre and post measurement is < 1, and the correlation between pre and (post $-$ pre) is generally negative -- it follows that if the treatment (your group allocation) as measured by raw scores happens to be an unfair disadvantage compared to control, it will have an unfair advantage with change scores.
  • The variance of the estimator used in ANCOVA is generally lower than that for raw or change scores (unless correlation between pre and post equals 1).
  • If the pre/post relationships differ between the two groups (slope), it is not as much of a problem than for any other methods (the change scores approach also assumes that the relationship is identical between the two groups -- the parallel slope hypothesis).
  • Under the null hypothesis of equality of treatment (on the outcome), no interaction treatment x baseline is expected; it is dangerous to fit such a model, but in this case one must use centered baselines (otherwise, the treatment effect is estimated at the covariate origin).

I also like Ten Difference Score Myths from Edwards, although it focuses on difference scores in a different context; but here is an annotated bibliography on the analysis of pre-post change (unfortunately, it doesn't cover very recent work). Van Breukelen also compared ANOVA vs. ANCOVA in randomized and non-randomized setting, and his conclusions support the idea that ANCOVA is to be preferred, at least in randomized studies (which prevent from regression to the mean effect).

share|improve this answer

Daniel B. Wright discusses this in section 5 of his article Making Friends with your Data. He suggests (p.130):

The only procedure that is always correct in this situation is a scatterplot comparing the scores at time 2 with those at time 1 for the different groups. In most cases you should analyse the data in several ways. If the approaches give different results ... think more carefully about the model implied by each.

He recommends the following articles as further reading:

  • Hand, D. J. (1994). Deconstructing statistical questions. Journal of the Royal Statistical Society: A, 157, 317–356.
  • Lord, F. M. (1967). A paradox in the interpretation of group comparisons. Psychological Bulletin, 72, 304–305. Free PDF
  • Wainer, H. (1991). Adjusting for differential base rates: Lord’s paradox again. Psychological Bulletin, 109, 147–151. Free PDF
share|improve this answer
1  
(+1) Thanks for these additional references. –  chl Dec 8 '10 at 8:21

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.