0
$\begingroup$

This is a pretest-posttest design with a control group (no treatment between measurements) and an experimental group (treatment between measurements).

All measurements consist of the same rubric and provide ordinal data (1,2,3,4).

Tests for metric data like t-tests are not appropriate. Many websites suggest using the Mann-Whitney U test as a non-parametric test for independent groups, with the difference between pretest and posttest score, i.e, gain score. My problem with these suggestions is that with ordinal data calculations like differences are not possible.

What test would be appropriate for this pre-test/post-test design?

Edit 2: Calculating differences is not possible for ordinal data

Pitouille write about "calculate the difference between post and pre-test". In my understanding, it is not possible to determine the difference with ordinal data, as it is not defined at all. If I could calculate the difference, the data would have to be at least interval scaled.

The difference corresponds to the usual gain score, which I thought of first. But because of the above reasoning, it is not possible to form the gain score.

Hence my question is: How can I determine the effects in pretest-posttest groups if I cannot calculate differences?

Edit 1: Possible examples of data.

First to be mentioned, the study is in the planning phase and I have not yet collected any data. Each group is expected to have 15 to 20 participants.

Candidate Control(0) or Experiment(1) group? Pretest-Score Posttest-Score
A 0 1 1
B 0 1 2
C 0 2 2
D 1 2 3
E 1 2 2
F 1 3 4
Improve this question
$\endgroup$
4
  • $\begingroup$ To understand better your problem, it would be helpful if you can share examples of the data you have to deal with. $\endgroup$
    – Pitouille
    Commented Sep 3, 2021 at 5:08
  • $\begingroup$ @Pitouille Thank you for your quick reply! If the examples attached are not sufficient, please let me know. $\endgroup$
    – Svana
    Commented Sep 3, 2021 at 7:27
  • $\begingroup$ I feel your question is more about structuring your test... did I get it right? $\endgroup$
    – Pitouille
    Commented Sep 3, 2021 at 12:48
  • 1
    $\begingroup$ Calculating differences between ordinal data (if expressed as numbers) is for sure possible, the question is rather whether this is meaningful. This is a very controversial issue. Even if in the application is question the meaning of numerical differences is not clear, it means at least the count of possible options between the values. And note that statistical model assumptions are not in terms of the meaning of the numbers, but in terms of their distributions. You can find a significant difference between distributions regardless of being sure what it means. $\endgroup$
    – Christian Hennig
    Commented Oct 21 at 10:13

2 Answers 2

Reset to default
3
$\begingroup$

Just as the rank difference test has made the Wilcoxon signed-rank test obsolete, the generalization of the rank difference test using the proportional odds model, either with random effects to handle the pairing, or using the cluster sandwich covariance matrix correction, will give you what you need. This is detailed here. The rank difference test and the PO model do not require you to subtract an ordinal variable from another ordinal variable.

Improve this answer
$\endgroup$
2
  • 1
    $\begingroup$ (+1) To do a minor clarification, for people searching this because there are a lot of "rank difference test" results when googling: this refers to Kornbrot's RDT. (Please correct me if I am wrong Frank) $\endgroup$
    – usεr11852
    Commented Oct 27 at 20:10
  • 1
    $\begingroup$ Correct - I need to give her credit every time I mention 'rank difference test'. $\endgroup$
    – Frank Harrell
    Commented Oct 28 at 22:46
0
$\begingroup$

I might miss the point... but at least it could contribute to push your question forward!

My understanding is that you are looking for the approach to conduct your test rather the test itself.

Supposing that the assumptions to conduct a Mann-Whitney U test are met, the approach would be to split your data in 2 samples (control and experiment) and for each candidate, calculate the difference between post and pre test.

Control

Canditate Post - Pre
A 0
B 1
C 0

Experiment

Canditate Post - Pre
D 1
E 0
F 1

... and then perform your test with you two independent samples.

EDIT:

So, I did miss the point :-)

There is study from Winter and Dodou that you can find in the stats literature that could be relevant to your problem: https://blog.minitab.com/en/adventures-in-statistics-2/best-way-to-analyze-likert-item-data-two-sample-t-test-versus-mann-whitney

You can also have a look at @Glen_b's answer: Mann Whitney or two tailed t-test

Improve this answer
$\endgroup$
1
  • $\begingroup$ Formerly, that was my idea too. In the post with the question, I added my concern as it was too long for the comment. $\endgroup$
    – Svana
    Commented Sep 3, 2021 at 16:52

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.