1
$\begingroup$

Quick question that I hope someone can help with. Imagine you have data from a randomized controlled trial (RCT) and you are interested in examining the impact of the treatment on a test score. Say that you administer the same test to participants prior to the treatment and after the treatment. I have presented means and standard deviations for each group in the table below. The control group is scoring a little higher on the pre-treatment test (but the difference between the control and treatment is not statistically significant. i.e., we are confident that the treatment indicator is an exogenous variable). We then examine the test scores after the treatment and find that the treatment effect is not statistically significant. However, if we take first differences, we find that the treatment effect is statistically significant. Which results should one trust? The endline only analysis or the first-difference analysis?


                | Treatment   |  Control   | H0: mean(Treatment) - mean(Control) = 0
  ------------------------------------------------------------------------------------   
  Before        |   60.61     |   63.41    |  Fail to reject
                |  (16.13)    |  (17.86)   |
  After         |   64.02     |   62.80    |  Fail to reject
                |  (11.42)    |  (14.76)   |
After - Before  |    3.41     |   -0.61    |  Reject            
                |  (13.99)    |  (15.10)   | 

The three test that are being carried out are :
1) $Treatment_{pre}$ vs $Control_{pre}$ (pre-treatment scores). This is to check if the groups are equivalent prior to receiving treatment.
2) $Treatment_{post}$ vs $Control_{post}$ (post-treatment scores). This is to check if the treatment has had an effect.
3) $(Treatment_{post} – Treatment_{pre})$ vs $(Control_{post} – Control_{pre})$. This is an alternative way of checking if the treatment is having an effect.

Given that we fail to reject the null in test (1), I'm wondering whether test (2) or test (3) is more appropriate for testing whether the treatment is having an effect?

$\endgroup$
3
  • $\begingroup$ It is unclear what you are testing in the second line. Exactly what is being compared to what? Treatment after vs treatment before? Treatment after vs. control after? Something else? $\endgroup$
    – whuber
    Apr 16, 2014 at 16:15
  • $\begingroup$ The fact that you fail to reject the null in test (1) does not mean that the null is true so I don't think test (2) is appropriate. $\endgroup$
    – Aghila
    Apr 17, 2014 at 10:32
  • $\begingroup$ You should never test for differences in pre-treatment/pre-randomization scores as per definition any difference is due to chance. Test 2 is asymptotically unbiased and therefore valid but you lose power due to small random baseline differences. Test 3 reduces the problem but is still not recommended. You should use an ANCOVA with treatment and baseline value of outcome as independent variables (and perhaps treatment x baseline interaction), see ncbi.nlm.nih.gov/pmc/articles/PMC1121605 or google Stephen Senn $\endgroup$
    – Daniel
    Feb 24, 2020 at 14:38

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.