A collegaue and I are conducting a pilot study in a school, the aim of which is to assess whether an "alternative" educational tool is more or less effective than traditional teaching methods. Students in each participating class were randomly assigned into two groups. At the start of the experiment (each class was tested at a separate time due to organizational constraints), every student filled out a pretest, then the groups were separated. One group participated in a traditional lesson based on the material in the test, while the other group participated in an alternative lesson in a different room with no contact between groups. Following the lessons, students filled out the same test again, with questions being ordered differently. I would like to run some t-tests on the difference between the pretest and the posttest between the two groups to see if there is a difference in knowledge gained.
Even though the assignment of students into groups was completely random (I used an online RNG tool to randomize numbers corresponding to the number of students in each class, and assigned the respective students from a list to numbers), the difference in pretest scores between the two groups is significant, namely, students assigned to the "traditional lesson" group have significantly higher pretest scores at baseline. The allocation of each student into a group was only revealed after the pretest, and students were allowed no communication during the experiment, so I have a hunch that this is a statistical anomaly due to small sample size (35 per group so far).
My question therefore is: Does this invalidate the inferences drawn from comparing the score differences across the two groups? If yes, what could be done to remedy it? I thought about adding the baseline (pretest) score for each student as a covariate in a GLM, but I am not certain it would help.
Thank you in advance!