What is the best analytic approach for two arm randomized pre-post trial?

Question

Study design is a classic pre-post design:

1. two trt arms (treatment, control). Subjects are randomly assigned into two arms.
2. outcome is continuous and measured at baseline and a follow-up time point (pre-score and post-score)
3. We would like to know if treatment works better than control

Statistical methods:

1. ANOVA on post-score or regular two sample t test: model: post-score=trt_arm
2. ANOVA on change score. change score=post-score - pre-score: model: change score=trt_arm
3. ANCOVA on post-score: model: post-score=trt_arm pre-score
4. Repeated measurement model on (pre-score, post-score) model: (pre-score,post-score)=time, trt_arm, trt_arm*time

Questions:

1. Which method is the best? Some literatures say it is ANCOVA and some literatures say it is repeated measures model
2. What does each method target? I have read over different literatures. what really confuses me is that it appears each method target something different. some examples:

ANOVA: whether the post-test means, differ between the two groups.

ANCOVA: whether the post-test means, adjusted for pre-test scores, differ between the two groups.

Repeated measures model: whether the mean change in the outcome from pre to post differed in the two groups.

Change score: whether the change score differ between the two groups.

3. If these different methods target different treatment effects and answer different questions, why do we need to put them in the same basket and compare them (in some literatures) ? Are they even comparable to each other?

4. Do we need to consider an interaction term in ANCOVA?

Answer 1

First, in randomized trials, all these methods eventually aim to estimate the same quantity: difference in post-treatment scores between two arms (our treatment effect), and the corresponding point estimators of treatment effect are all unbiased. This definition of treatment effect makes sense if you know pre-score is just a baseline covariate (not outcome) like other covariates. If you do not collect any baseline variables, what do you compare between two arms? Difference in post-treatment scores. We normally use ANOVA or two sample t.

So these methods are comparable and we can compare the variances of their treatment effect estimators and choose the method that yields the smallest variance estimator for its treatment effect estimator.

Second, it turns out ANCOVA main effect model is the most efficient approach with smallest variance estimator. That is, you fit a regular OLS regression in SAS

Proc reg;
     Model Post_score=group pre_score;
     Run;

And you read the estimate and p-value associated with group variable. If you really want to repeated measures model, you should try a constrained repeated measure model (cRM) in which the main effect term for group should be excluded

Proc mixed;
     Model Y=time time*group;
     Repeated time/subject=subj;
     Run; 

You report the estimate and p-value associated with “time*group”.

The reason why regular repeated measure model (RM) is less efficient is because it needs to estimate one extra parameter for the main effect of treatment group as follows:

Proc mixed;
     Model Y=group time time*group;
     Repeated time/subject=subj;
     Run; 

You report the estimate and p-value associated with “time*group”. With the inclusion of “group”, RM assumes the mean pre-scores are different between two arms, which is generally not true in a randomized trial. That is why cRM forces the coefficient associated with group to be zero and remove this main effect group term from the model.

cRM and ANCOVA estimators of the treatment effect are equivalent, their variances are equivalent, and thus they have comparable performance. However, ANCOVA is more meaningful conceptually because pre-score is really a baseline covariate, not really an outcome. Outcome is something we observe after applying the treatment.

Change score is also less efficient. We fit a change score model in SAS like this:

Proc reg;
     Model change_score=group;
     Run;

We report the estimate and p-value associated with “group”. Change score model and regular RM model have the exactly the same point estimator of treatment effect. When there is no missing data, the two estimates are numerically same. Thus, both change score model and regular RM do not utilize the fact that pre-scores are the same between two arms.

ANOVA or two sample t test on post- score basically ignores the baseline information so it is least efficient in general situations.

Third, if there is an interaction effect. The situation is a bit more complicated. In previous discussion, we basically assume the pattern of association between pre-post scores is the same for everybody. When there is an interaction effect, it suggests the pattern of association between pre-post scores can be different between two arms. For example, subjects in the treatment group may respond very differently to treatment so their post-scores can vary a lot and the association between pre-post scores is generally weaker (than their counterparts in control group). We normally would expect the association between pre- and post-scores are stronger because placebo control would not change post-scores that much.

In this case, you can either fit an ANCOVA interaction model.

Proc reg;         
     Model Post_score=group pre_score_meancentered group*pre_score_meancentered;
     Run;

pre_score_meancentered is computed by subtracting overall sample mean pre-score from each individual pre-score. You can get the estimate associated with group. However, p-value is not correct (heteroscedasticity and extra variability in estimating sample mean are not accounted for). There is an adjustment in variance estimator you need to do.

Or you can still fit an ACNOVA main effect model

Proc reg;
     Model Post_score=group pre_score;
     Run;

The main effect model is still OK even we need to fit an interaction term. The estimate associated with “group” is still unbiased. When the design is balanced with equal sample size in both arms, regular p-value is valid. In the unbalanced design, regular p-value is not correct. You need to use heteroscedasticity consistent variance (reported from SAS proc reg).

The two ANCOVA models are the same in balanced design. When design is unbalanced, ANCOVA interaction model performs slightly better with smaller variance estimate and smaller p-value. However, ANCOVA main effect is much simpler to implement. When sample size is relatively large, there is not much difference between two ANCOVA models.

Certainly, you can fit a cRM model with difference variance/covariance structure specified for each treatment arm, if you really like to treatment baseline covariate pre-score as an outcome. Anyway, It will perform comparably to ANCOVA interaction model.

So final recommendation is: use ANCOVA. It is easy to implement and it is the most efficient approach. Do heteroscedasticity check if there is a potential interaction effect. If so, use homoscedasticity consistent variance and associated p-value when design is unbalanced.

Some references:

Senn S. Change from baseline and analysis of covariance revisited.

Kenward MG, White IR, Carpenter JR. Re: should baseline be a covariate or dependent variable in analyses of change from baseline in clinical trials? (Liu GF et al., Stat Med 2009; 28: 2509–30).

Wan.f Statistical analysis of two arm randomized pre-post design with one post-treatment measurement