1
$\begingroup$

Is it possible to assess the effectiveness of a classroom intervention by:

  1. Taking scores from an entrance exam administered before the intervention
  2. Ranking these scores (i.e. 1st, 2nd, 3rd, ...)
  3. Providing students with an optional learning app (the intervention)
  4. Administering a final year exam
  5. Ranking this final year exam
  6. Using the directional change in rank (i.e. increased or decreased) to assess whether the learning app had a significant impact

If this is a valid experiment, what statistical methods should I use?

$\endgroup$
0
$\begingroup$

After considering this further, I feel that ranking items is a bad way to "normalize" the scores in these two tests for a number of reasons:

  1. Ranking creates unwanted dependencies between participants. For example, imagine a scenario where you have 10 students undergoing this experiment. Then imagine that all 10 students performed equally well on the second test as they did on the first. This would result in all 10 students having the same rank on both tests (i.e. 1st -> 1st, 2nd -> 2nd etc.). However, if you imagine the same situation again, only now, with the person who was ranked 1st, ranking 10th on the second test (i.e. 1st -> 10th). What we'd see is 9 out of the 10 students increase their rankings (i.e. 2nd -> 1st, 3rd -> 2nd etc.) and only 1 decrease in ranking. This 1 person flunking would make it appear that 9/10 had improved while, in reality, they had only performed equally well.

  2. Ties will almost certainly be different between tests which will also lead to the problem described above. For example, imagine there are no ties on the first test but then, in the second test, the person who was initially ranked 2nd gets the same score as the person who was initially ranked 1st. Even if all other students performed equally well on both tests, the "dense" ranking system would show that everyone ranked below 2nd had improved and increased in rank. Of course this specific example can be fixed by using a different ranking system, however, the other systems still introduce similar problems in different scenarios.

  3. We lose information by ranking. For example, imagine the person who ranked 1st in the first test received a test score that was 50% higher than the person who was ranked 2nd. Then in the second test the person who was ranked 2nd performed extremely well and closed this 50% gap down to 5%. Looking at the rankings alone, we would miss this 45% improvement!

I've found that the best way to analyse this is to use the original scores without ranking. Essentially, group the students into two groups (students who took advantage of the optional intervention and those who didn't). We drew this line arbitrarily at 80% engagement rate (i.e. answering 80% of the optional questions).

We then take this information and run a t-test to check that we have a significant difference in means between these two groups. This gives us a general idea if there appears to be a significant difference.

In R this would look like:

my.t.test <- t.test(FINALEXAM ~ GROUP, data = my.data)
my.t.test # Show results of the t-test

Then to make things more robust you can use ANCOVA controlling for the entrance exam as "baseline academic ability".

Again, to do this in R you can simply use:

my.anova <- aov(FINALEXAM ~ PRETEST + GROUP, data = my.data)
summary(my.anova) # Show results of the ANOVA
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.