3
$\begingroup$

A collegaue and I are conducting a pilot study in a school, the aim of which is to assess whether an "alternative" educational tool is more or less effective than traditional teaching methods. Students in each participating class were randomly assigned into two groups. At the start of the experiment (each class was tested at a separate time due to organizational constraints), every student filled out a pretest, then the groups were separated. One group participated in a traditional lesson based on the material in the test, while the other group participated in an alternative lesson in a different room with no contact between groups. Following the lessons, students filled out the same test again, with questions being ordered differently. I would like to run some t-tests on the difference between the pretest and the posttest between the two groups to see if there is a difference in knowledge gained.

Even though the assignment of students into groups was completely random (I used an online RNG tool to randomize numbers corresponding to the number of students in each class, and assigned the respective students from a list to numbers), the difference in pretest scores between the two groups is significant, namely, students assigned to the "traditional lesson" group have significantly higher pretest scores at baseline. The allocation of each student into a group was only revealed after the pretest, and students were allowed no communication during the experiment, so I have a hunch that this is a statistical anomaly due to small sample size (35 per group so far).

My question therefore is: Does this invalidate the inferences drawn from comparing the score differences across the two groups? If yes, what could be done to remedy it? I thought about adding the baseline (pretest) score for each student as a covariate in a GLM, but I am not certain it would help.

Thank you in advance!

$\endgroup$
2
  • $\begingroup$ What is your research question ? $\endgroup$ Sep 21, 2020 at 21:36
  • $\begingroup$ I would like to know if the alternative educational method has lower, the same, or higher efficiency in improving scores from the pretest to the posttest. Essentially, I'm interested in whether the difference in scores will be significantly higher or lower. $\endgroup$ Sep 22, 2020 at 7:02

2 Answers 2

2
$\begingroup$

The randomization is not supposed to balance differences between groups. Since the assignment is random, it will happen 5% of the times that the differences will be significant with the alpha =0.05. The randomization is there to get rid of systematic biases and unknown unmeasured effects, on average. Yes, it's possible that your results are due to chance, but since the assignment was completely random, we can calculate what is the probability that if there is no effect the results would happen by chance (i.e. p-value). So your study is still valid because it's the randomness of the assignment that makes the results valid, not the balance after assignment.

Assuming the effect of your treatment is constant for everybody, then pretest scores won't affect the change scores you are testing. In general, testing post-test scores adjusted by pre-test scores is the recommended approach, because it is more sensitive and has fewer assumptions. If you think pre-test scores affect the post-test scores, this is what you should do. Some people don't like it because they don't understand adjustment, but statistically, it's the correct thing to do. There is no difference between testing post-test scores adjusted by pretest scores, or testing change scores adjusted by pre-test scores.

See Vickers 2001 Analysing controlled trials with baseline and follow up measurements https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1121605/ this datacolada blog http://datacolada.org/39 Griffin et al 1999 https://deepblue.lib.umich.edu/handle/2027.42/73008 this thread Best practice when analysing pre-post treatment-control designs and much more

$\endgroup$
5
  • $\begingroup$ Your answer (+1) appeared while I was writing mine. Potentially useful references. $\endgroup$
    – BruceET
    Sep 21, 2020 at 21:31
  • $\begingroup$ Thanks! How would one go about adjusting post test scores by pretest scores? Using an ANCOVA and adding it as a covariate? Sorry if the question is silly, I am sometikmes confused by terminology. $\endgroup$ Sep 22, 2020 at 5:48
  • 1
    $\begingroup$ yes, exactly. you run ANCOVA with your posttest as an outcome variable and pretest scores and treatment as dummy variable as inputs, and you are checking if the treatment dummy is significant in this model $\endgroup$
    – rep_ho
    Sep 22, 2020 at 10:00
  • $\begingroup$ Thanks! It appears that after collecting some more data, the difference between the pretest scores of the two groups are no longer significant when checked by an independent t-test. I nonetheless ran an ANCOVA as you described, and both the group dummy and the pretest covariate are significant predictors, but their interaction is not. How should I interpret this result? Thanks in advance! $\endgroup$ Sep 22, 2020 at 17:48
  • $\begingroup$ Good. I wouldn't include the interaction term. You can interpret it as a significant effect of your treatment (since the dummy is significant), adjusted for the pre-test variables. The difference between treatment and control is the coefficient of your dummy variable. Check if it is in the right direction. A significant effect of the pre-test scores is expected, but the results would be valid even if it's not significant. If you include interaction, then the interpretation becomes harder. The interaction is testing the difference between slopes in your treatment/control groups $\endgroup$
    – rep_ho
    Sep 23, 2020 at 13:42
2
$\begingroup$

Presumably, it is the average improvement made by each group that matters. So, you can find the post-test minus pre-test differences $D_i$ for each student. Then do a 2-sample t test on the two groups of $D_i$s.

Because the two groups differed on average as to pre-test scores, it seems possible that the pre-test scores for one group may also have had a higher variability than the other. So it seems best to use Welch t tests which would not be invalidated by any difference in variabilities of post minus pre $D_i$s between groups.

Note: It is unusual for two randomized groups to differ significantly at the 5% level, but that happens (by definition) 5% of the time, as shown in the simulation below, and if you look at $D_i$s this should be no problem.

set.seed(921)
m = 10^5;  pv = numeric(m)
for (i in 1:m) {
 x = rnorm(70, 100, 15);  xr = sample(x)
 pv[i] = t.test(xr[1:35], xr[36:70])$p.val
}
mean(pv <= 0.05)
[1] 0.05091
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.