I have 3 groups randomised to intervention A or B or control. Participants were measured pre and post intervention - should I use repeated measures ANOVA or ANCOVA with the variable measure at time 1 (pre) as the covariate? Variables are continuous. thanks
$\begingroup$
$\endgroup$
2
-
$\begingroup$ Sheila, What did you go with in the end? I am currently trying to decide the exact same thing (only with a 2x2 design, experiment group vs. control group at Time1 and Time2). My numbers are small (n=19 in experiment and n=17 in control) so need to bear that in mind - plus the fact that at least one of the DVs is not normally distributed... Thanks! $\endgroup$– user13833Commented Sep 4, 2012 at 14:00
-
$\begingroup$ Highly relevant discussion with several insightful answers and further links: Best practice when analysing pre-post treatment-control designs $\endgroup$– amoebaCommented Feb 19, 2014 at 14:27
Add a comment
|
4 Answers
$\begingroup$
$\endgroup$
2
Actually, the post minus pre approach is one that you should always try to avoid. The issue is not about statistics. It is about measurement. When you calculate difference scores, the reliability of the measures become very low. You don't want that.
I prefer ANCOVA if the assumptions are met (and it will in most cases), because the interpretation is easier. The readers of your study will have to look at only the post averages, 3 averages in your case, instead of all 6. If you get an interaction effect, which is expected that you do, you'd have to do a post hoc comparison of that effect. This will give you even more p-values that make reading your paper even harder.
If you want others to cite your study, make it easier to read.
-
2$\begingroup$ why would you say that the reliability is very low? by reliability i assume you are talking about the same reliability as referred in psychometrics. in that case, shouldn't the random error of the pre and post cancel each other out? i.e. it affects everyone randomly. $\endgroup$– RJ-Commented Aug 29, 2012 at 3:27
-
$\begingroup$ It is a pretty common knowledge that difference scores have low reliability. For example: jstor.org/discover/10.2307/… The issue may affect everyone equally, but unfortunately, statistical tests lose power as reliability becomes lower. In theory, if reliability is 0, even strong relationships will never be significant regardless of sample size. $\endgroup$– HotakaCommented Sep 21, 2013 at 21:34
$\begingroup$
$\endgroup$
6
I suspect the best approach would be to get difference scores (i.e., post - pre) for each experimental unit, and then run a simple one-way ANOVA.
answered Mar 25, 2012 at 15:00
-
1$\begingroup$ It may be the easiest but I don't think characterizing it as the "best" is reasonable. At the onset it precludes seeing if any baseline*treatment interaction exists, and as John mentions it is problematic if the baseline has a causal effect on the post-treament (independent of treatment itself). Paul Allison in this article gives some examples where change scores are preferable. $\endgroup$– Andy WCommented Mar 25, 2012 at 15:24
-
$\begingroup$ It is contentious for sure, see a few questions on the site of relevance. Best practice when analysing pre-post treatment-control designs, Should the difference between control and treatment be modelled explicitly or implicitly?, Is it valid to include a baseline measure as control variable when testing the effect of an independent variable on change scores?. $\endgroup$– Andy WCommented Mar 25, 2012 at 15:24
-
$\begingroup$ this is very helpful _ i have already done a repeated measure and have a sign interaction but contrasts did not reflect this . so i looked at a change score (ie i made the new variable of the difference) and put this in a simple ANOVA and it showed where the signficiant differences were but I am worried that this is poor statistical methodology and that all the testing should be within one test?? maybe this isnt right - i think on a quick look that ANCOVA may show this significance but dont want to inflate the type 1 error. i dont know if group is correlated to outcome - i'll look at this $\endgroup$– SheilaCommented Mar 25, 2012 at 15:48
-
$\begingroup$ @AndyW A lot of this discussion focuses on situations where people cannot be randomly assigned to the condition. Not to say that it's entirely irrelevant but that's something to keep in mind when trying to relate it to the present question. $\endgroup$– GalaCommented Jun 10, 2013 at 12:52
-
1$\begingroup$ Sure @GaëlLaurans, the refence I gave in my comment to the Paul Allison article provides data generating process examples where change scores are preferable. I disagree they should be a default though, that was my only point. $\endgroup$– Andy WCommented Jun 10, 2013 at 13:11
$\begingroup$
$\endgroup$
9
I would generally say that you need a repeated measures model with group, pre_post, and an interaction term. What you want to know is whether the post-test score is dependent upon the intervention, so you need to see an interaction. The meaning of that interaction would depend on actual scores.
A simpler method might be to use a pre_post subtraction score. That would make the assumption that pre-test differences (significant or otherwise), do not affect the outcome at all, or are not present at all. Check your groups, and if they are very highly similar then the subtraction score and one-way ANOVA is an easy thing to do.
The ANCOVA is much more like the difference scores or also an ANOVA on the residuals left after removing pre-test. It's not wrong, and some people prefer it. I believe I read a paper once recommending it but even they qualified it because pre-test may be correlated with group. In that case interpretation becomes difficult.
Why not run the full ANOVA and the ANCOVA? If they reach similar conclusions you're safe. If they reach different conclusions then think much more about what your data mean and maybe come back on here and ask for help at the interpretation stage.
-
-
$\begingroup$ I think that the F test for the interaction term in ANOVA with group and pre_post as factors yields exactly the same result than a one-way ANOVA on difference scores. Whatever the merits of this approach compared to an ANCOVA, it seems that it does not matter whether you choose for an independent sample t-test (in the two-group case) or one-way ANOVA on gain scores vs. a 2x2 mixed ANOVA. At the end of the day, comparing this and an ANCOVA seems like a good strategy however (+1). $\endgroup$– GalaCommented Jun 10, 2013 at 13:07
-
$\begingroup$ The interaction term and pre_post should be the same, yes. But, by failing to analyze the pre effects you've hidden potential interpretation issues. It's better to have them all clearly in the open. $\endgroup$– JohnCommented Jun 10, 2013 at 13:26
-
$\begingroup$ Well, if treatment assignment is properly randomized, it does not make sense to test for a difference. In any case, there is no additional assumption in the change score approach. If you are prepared to interpret the interaction term, you might just as well run the one-way ANOVA directly. $\endgroup$– GalaCommented Jun 10, 2013 at 15:14
-
1$\begingroup$ I'm not sure what you mean by "properly randomized". Random assignment can result in baseline differences by random chance and those can impact the interpretation of subsequent difference scores. $\endgroup$– JohnCommented Jun 10, 2013 at 20:12
$\begingroup$
$\endgroup$
ANCOVA gives the same results asymptotically as the contrained repeated measures model (no treatment main effect term). The usual repeated measures model including time, treatment, time*treatment terms assumes the baseline scores are not balanced so its estimator is less efficient.
