Quick question that I hope someone can help with. Imagine you have data from a randomized controlled trial (RCT) and you are interested in examining the impact of the treatment on a test score. Say that you administer the same test to participants prior to the treatment and after the treatment. I have presented means and standard deviations for each group in the table below. The control group is scoring a little higher on the pre-treatment test (but the difference between the control and treatment is not statistically significant. i.e., we are confident that the treatment indicator is an exogenous variable). We then examine the test scores after the treatment and find that the treatment effect is not statistically significant. However, if we take first differences, we find that the treatment effect is statistically significant. Which results should one trust? The endline only analysis or the first-difference analysis?
| Treatment | Control | H0: mean(Treatment) - mean(Control) = 0
------------------------------------------------------------------------------------
Before | 60.61 | 63.41 | Fail to reject
| (16.13) | (17.86) |
After | 64.02 | 62.80 | Fail to reject
| (11.42) | (14.76) |
After - Before | 3.41 | -0.61 | Reject
| (13.99) | (15.10) |
The three test that are being carried out are :
1) $Treatment_{pre}$ vs $Control_{pre}$ (pre-treatment scores). This is to check if the groups are equivalent prior to receiving treatment.
2) $Treatment_{post}$ vs $Control_{post}$ (post-treatment scores). This is to check if the treatment has had an effect.
3) $(Treatment_{post} – Treatment_{pre})$ vs $(Control_{post} – Control_{pre})$. This is an alternative way of checking if the treatment is having an effect.
Given that we fail to reject the null in test (1), I'm wondering whether test (2) or test (3) is more appropriate for testing whether the treatment is having an effect?