4
$\begingroup$

My colleague and I are professors and we conducted an experiment in which we would please like some advice on deciding which tests to use in SPSS.

He taught 4 classes of students on 2 different occasions and I taught a different group of 4 classes of students on 2 different occasions. I was the experimental group and taught all of my sessions incorporating an educational game. He was the control group and did not use any games. We each gave the students a pre-test at the beginning of the first session and a post-test at the end of the second class.

Our hypothesis is that students in the experimental (games) classes performed better on the post-test than students in the control group. Unfortunately we didn't think to assign each student a number so that we could figure out which pre and post-test belonged to who. So basically we have a ton of pre and post-tests divided by class but not by student. Is there a a way we could conduct statistical analyses for the groups instead of individuals to see if our hypothesis was concerned?

Improve this question
$\endgroup$
2
  • $\begingroup$ In the last paragraph, you state that your hypothesis was that students in the experimental group would perform better in the pre-test than students in the control group. Did you mean post-test? $\endgroup$
    – Patrick Coulombe
    Commented Mar 28, 2014 at 1:26
  • $\begingroup$ This is kind of tangential to your question, but one concern given your description is that the level of instructor is confounded with game/no-game. You don't know if your test is showing that the game improved things or you're a better instructor than the other. Make sure to indicate in your analysis that you're assuming there's negligible effect due to instructor. $\endgroup$
    – neverKnowsBest
    Commented Apr 12, 2014 at 15:07

2 Answers 2

Reset to default
1
$\begingroup$

If I were trying to answer this I would make a plot of the grades for the before into a single plot and after into a single plot. Consider the method shown here (link).

I would look at how central tendencies changed before vs. after. This includes the mean, median, and possibly mode. I would look at how variation changed before and after. This includes both standard deviation and inter-quartile range. I would look at how extremal values change before and after.

You can use sample sizes to estimate the uncertainty.

A good, first level, sort of test is to see if the +/- 95% confidence interval for the mean of the alternate treatment is within the +/- 95% confidence interval for the reference treatment. You could say that if this was not the case then the results of the alternate treatment are statistically significantly different than the reference treatment.

I would also consider the +/-95% confidence interval for the standard deviation of the alternate treatment versus the reference. Sometimes an approach gives better consistency, or pulls in the extremal values, even when the mean does not significantly change.

Improve this answer
$\endgroup$
3
  • $\begingroup$ Thanks so much for your help, EngrStudent! You wouldn't do any type of t-test, then? $\endgroup$
    – Kate
    Commented Mar 29, 2014 at 13:40
  • $\begingroup$ @Kate - That goes into the pile of "use sample sizes to estimate the uncertainty". Uncertainty estimation without accounting for sample size is like an airplane with an engine, but no wings. $\endgroup$
    – EngrStudent
    Commented Mar 31, 2014 at 19:12
  • 2
    $\begingroup$ @EngrStudent you might need to explain what you mean by "sample size" in a correlated data analysis. A design which consists of 100 people measured twice has indeed two hundred rows of data. In this case of missing linkage, the OP would know that there are N=100 clusters. Perhaps instead of accounting for sample size, you mean to say, accounting for dependence. $\endgroup$
    – AdamO
    Commented Jul 18, 2023 at 4:39
1
$\begingroup$

Well, our intuition would lead us to believe this is a tragic loss of information. Simulations and designs would corroborate this belief. If you lose the link IDs to join correlated records of data in a design such as this one, it is not possible to conduct, for instance, the oft recommended ANCOVA design for a difference in difference analysis in a design like this. But it behooves us to ask: what happens if we run an independent data analysis?

In other words, the hypothesis that you are trying to test is this: $$ \mathcal{H}_0: \mu_{2,2}-\mu_{2,1} = \mu_{1,2} - \mu_{1,1}$$

Where $\mu_{i,j}$ represents data from the $i$-th group measured at time $j$. Whereas a correlated data analysis would have given us a more precise estimate of $\mu_{i,2}-\mu_{i,1}$, you can still calculate these quantities separately, such as in a regression, and calculate the difference in difference for a significance test without accounting for data correlation. One can show this leads to a slightly less efficient analysis, but the test is still powered to detect the effect. The worst case scenario is that the test does not produce a significant finding, because you can't be assured that the nominal alpha level has truly been achieved as you are not accounting for dependence between observations.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.