My colleague and I are professors and we conducted an experiment in which we would please like some advice on deciding which tests to use in SPSS.

He taught 4 classes of students on 2 different occasions and I taught a different group of 4 classes of students on 2 different occasions. I was the experimental group and taught all of my sessions incorporating an educational game. He was the control group and did not use any games. We each gave the students a pre-test at the beginning of the first session and a post-test at the end of the second class.

Our hypothesis is that students in the experimental (games) classes performed better on the post-test than students in the control group. Unfortunately we didn't think to assign each student a number so that we could figure out which pre and post-test belonged to who. So basically we have a ton of pre and post-tests divided by class but not by student. Is there a a way we could conduct statistical analyses for the groups instead of individuals to see if our hypothesis was concerned?

  • $\begingroup$ In the last paragraph, you state that your hypothesis was that students in the experimental group would perform better in the pre-test than students in the control group. Did you mean post-test? $\endgroup$ – Patrick Coulombe Mar 28 '14 at 1:26
  • $\begingroup$ This is kind of tangential to your question, but one concern given your description is that the level of instructor is confounded with game/no-game. You don't know if your test is showing that the game improved things or you're a better instructor than the other. Make sure to indicate in your analysis that you're assuming there's negligible effect due to instructor. $\endgroup$ – neverKnowsBest Apr 12 '14 at 15:07

If I were trying to answer this I would make a plot of the grades for the before into a single plot and after into a single plot. Consider the method shown here (link).

I would look at how central tendencies changed before vs. after. This includes the mean, median, and possibly mode. I would look at how variation changed before and after. This includes both standard deviation and inter-quartile range. I would look at how extremal values change before and after.

You can use sample sizes to estimate the uncertainty.

A good, first level, sort of test is to see if the +/- 95% confidence interval for the mean of the alternate treatment is within the +/- 95% confidence interval for the reference treatment. You could say that if this was not the case then the results of the alternate treatment are statistically significantly different than the reference treatment.

I would also consider the +/-95% confidence interval for the standard deviation of the alternate treatment versus the reference. Sometimes an approach gives better consistency, or pulls in the extremal values, even when the mean does not significantly change.

  • $\begingroup$ Thanks so much for your help, EngrStudent! You wouldn't do any type of t-test, then? $\endgroup$ – Kate Mar 29 '14 at 13:40
  • $\begingroup$ @Kate - That goes into the pile of "use sample sizes to estimate the uncertainty". Uncertainty estimation without accounting for sample size is like an airplane with an engine, but no wings. $\endgroup$ – EngrStudent - Reinstate Monica Mar 31 '14 at 19:12

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.