Consider an experiment where randomly selected experts from a set are asked to attribute Likert-scale ratings to randomly selected objects from two groups A and B. Now, suppose I want to test whether there is a statistically significant difference between the distributions of ratings for objects in groups A and B. Rating distributions are strongly non-normal.
My first thought would have been to use Mann-Whitney U but this experimental design violates multiple independence assumptions. Within a single group, multiple ratings may come from the same expert OR refer to the same object. Between groups, some experts may have rated objects both in group A and B.
What is the statistically correct approach to perform this test? I can think of several ways to proceed:
- Use a different approach which does not require independence (if so, which one? Permutation test?)
- Perform a Mann-Whitney test on a subset of ratings (e.g., ensure that the set of experts for group A and B is disjoint, and each expert only rated one object per group)
- Give up
- Something else (mixed effect model, etc.)?