1
$\begingroup$

I have conducted a controlled experiment with the goal to determine whether an educational intervention improves the students' performance and self-assessment of their skills. Students from the same course performed a pre-test that on the one hand produces a score and their own assessment of their skills (in a category from 1- High skill level to 5 - No skill level). Then the students were separated into a control group with an alibi educational intervention and an experimental group that received the "real" educational intervention. The post-test was performed later and was identical to the pre-test.

Now my question is how to test whether there was a positive shift in self-assessment after the interventions and whether the shift was higher/lower for the experimental group in comparison to the control group.

From my research, I would assume that since the dependent variable is a categorical variable, i.e. the skill level, I would need something like a Chi-square test or Wilcoxon signed rank test. The former tests if there is a difference in the distributions while the latter tests if the values are higher/lower.

So would the Wilcoxon signed rank test be the appropriate choice? I.e. I test if there is a shift in the control group and then I test if there is a shift in the experimental group and compare the results.

$\endgroup$

1 Answer 1

1
$\begingroup$

One important question here is how this step was done:"[...] students were separated into a control group [...] and an experimental group [...]". If this was done using individual randomization, then the kind of analyses I describe apply. If not (e.g. decided by teacher according to their personal preference) other methods will be needed that account for the group assignment mechanism (e.g. propensity score methods). Your life will be a lot easier, if you did a randomized experiment (disclaimer: I'll assume you did in what follows, but naively using methods for randomized experiments when you did not randomize would be completely wrong).

The outcome you have seems to be an ordered categorical (aka ordinal) variable rather than an unordered categories. From that perspective, I might be tempted to analyze this using ordinal regression. There's a number of nice software packages for that kind of model, e.g. the brms R package is used in this tutorial. Besides the group assignment, obviously the pre-intervention assessment would be another key covariate. It would probably be too simplistic to assume it would have a linear effect across its numerical levels, but treating it as e.g. a monotonic effect might be reasonable (again, a lot of software supports this, e.g. here's a vignette for brms).

Alternatively, you can of course do a rank test, which (to take into account the pre-intervention assessment) you could stratify by the pre-intervention assessment outcome. However, for moderate sized datasets you will struggle a little to get decent power (or with empty strata) and its harder to guess whether the stratification would help or harm (if the sample size is too small it may hurt, if its larger it will help, but it's difficult to guess where the flipping point would be). Additionally, getting an easily interpreted effect size estimate that is aligned with the test will be a real struggle (e.g. a Hodges-Lehmann estimate is a bit of a nuisance to interpret and should not be a confused with the difference in medians).

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.