Significant difference between randomized groups at baseline?

Question

A collegaue and I are conducting a pilot study in a school, the aim of which is to assess whether an "alternative" educational tool is more or less effective than traditional teaching methods. Students in each participating class were randomly assigned into two groups. At the start of the experiment (each class was tested at a separate time due to organizational constraints), every student filled out a pretest, then the groups were separated. One group participated in a traditional lesson based on the material in the test, while the other group participated in an alternative lesson in a different room with no contact between groups. Following the lessons, students filled out the same test again, with questions being ordered differently. I would like to run some t-tests on the difference between the pretest and the posttest between the two groups to see if there is a difference in knowledge gained.

Even though the assignment of students into groups was completely random (I used an online RNG tool to randomize numbers corresponding to the number of students in each class, and assigned the respective students from a list to numbers), the difference in pretest scores between the two groups is significant, namely, students assigned to the "traditional lesson" group have significantly higher pretest scores at baseline. The allocation of each student into a group was only revealed after the pretest, and students were allowed no communication during the experiment, so I have a hunch that this is a statistical anomaly due to small sample size (35 per group so far).

My question therefore is: Does this invalidate the inferences drawn from comparing the score differences across the two groups? If yes, what could be done to remedy it? I thought about adding the baseline (pretest) score for each student as a covariate in a GLM, but I am not certain it would help.

Thank you in advance!

Answer 1

The randomization is not supposed to balance differences between groups. Since the assignment is random, it will happen 5% of the times that the differences will be significant with the alpha =0.05. The randomization is there to get rid of systematic biases and unknown unmeasured effects, on average. Yes, it's possible that your results are due to chance, but since the assignment was completely random, we can calculate what is the probability that if there is no effect the results would happen by chance (i.e. p-value). So your study is still valid because it's the randomness of the assignment that makes the results valid, not the balance after assignment.

Assuming the effect of your treatment is constant for everybody, then pretest scores won't affect the change scores you are testing. In general, testing post-test scores adjusted by pre-test scores is the recommended approach, because it is more sensitive and has fewer assumptions. If you think pre-test scores affect the post-test scores, this is what you should do. Some people don't like it because they don't understand adjustment, but statistically, it's the correct thing to do. There is no difference between testing post-test scores adjusted by pretest scores, or testing change scores adjusted by pre-test scores.

See Vickers 2001 Analysing controlled trials with baseline and follow up measurements https://www.ncbi.nlm.nih.gov/pmc/articles/PMC1121605/ this datacolada blog http://datacolada.org/39 Griffin et al 1999 https://deepblue.lib.umich.edu/handle/2027.42/73008 this thread Best practice when analysing pre-post treatment-control designs and much more

Answer 2

Presumably, it is the average improvement made by each group that matters. So, you can find the post-test minus pre-test differences $D_i$ for each student. Then do a 2-sample t test on the two groups of $D_i$s.

Because the two groups differed on average as to pre-test scores, it seems possible that the pre-test scores for one group may also have had a higher variability than the other. So it seems best to use Welch t tests which would not be invalidated by any difference in variabilities of post minus pre $D_i$s between groups.

Note: It is unusual for two randomized groups to differ significantly at the 5% level, but that happens (by definition) 5% of the time, as shown in the simulation below, and if you look at $D_i$s this should be no problem.

set.seed(921)
m = 10^5;  pv = numeric(m)
for (i in 1:m) {
 x = rnorm(70, 100, 15);  xr = sample(x)
 pv[i] = t.test(xr[1:35], xr[36:70])$p.val
}
mean(pv <= 0.05)
[1] 0.05091