Can I use Mann-Whitney U test for within group analysis?

Question

I am conducting a within-group study where participants rate the perceived helpfulness of ideas on a Likert scale (DV) across two different days (Day 1 and Day 2), serving as the independent variable (IV).

The challenge I'm encountering is that each participant rated a different set of ideas on each day, and not every participant rated ideas on both days. As a result, the ratings cannot be directly paired by participants across the two days. For example:

| day 1 | day 2 |

| p1's rating on idea#1 | p3's rating on idea#5 |

| p2's rating on idea#3 | p3's rating on idea#2 |

...

Given that the data is not normally distributed, my initial thought was to use the Wilcoxon Signed-Rank test to analyze the differences between Day 1 and Day 2. However, I realized this may not be appropriate due to the lack of paired data. Would this be the case where the Mann-Whitney U test is acceptable?

Answer 1

1. The fact that the sample data is not normally distributed is not your biggest problem. But your data is ordinal (very helpful, somewhat helphul, indiferent, somewhat unhelpful, very unhelpful), and not interval or ratio. So you can not use any numerical method (normal or not), compute any means, etc. (comment: if you had used a VAS -visual analog scale-, you could have computed means etc., and used normal methods... Food for thoughts for the next study).
2. Your data is not paired: so you can not use a paired Wilcoxon signed rank test (or even a paired sign test).
3. Could you use a Mann-Whitney U test. Yes. But... What are you trying to prove? What will be your null hypothesis? The "best" (I would argue the only valid null, but that is a discussion for another day) null for MWUt is that the probability that the helpfulness was perceived as greater on day 1 than on day 2 is equal to 0.5. P(Day1>Day2)=0.5. MWUt is a test of stochastic superiority (helpfulness was perceived as significantly higher on one of the days), and not a test of medians (as too often described).

So as long as you interpret it as a test of stochastic superiority, MWUt is fine, and maybe one of the only few choices you have.

A possible alternative is to use a 2x5 (I assume 5 level Likert?) contingency matrix, and run a test (Fisher, Chi2) to see if the oberved proportions are compatible with a null of "all the 5 proportions are the same on both days).