Consider an experiment where randomly selected experts from a set are asked to attribute Likert-scale ratings to randomly selected objects from two groups A and B. Now, suppose I want to test whether there is a statistically significant difference between the distributions of ratings for objects in groups A and B. Rating distributions are strongly non-normal.

My first thought would have been to use Mann-Whitney U but this experimental design violates multiple independence assumptions. Within a single group, multiple ratings may come from the same expert OR refer to the same object. Between groups, some experts may have rated objects both in group A and B.

What is the statistically correct approach to perform this test? I can think of several ways to proceed:

  1. Use a different approach which does not require independence (if so, which one? Permutation test?)
  2. Perform a Mann-Whitney test on a subset of ratings (e.g., ensure that the set of experts for group A and B is disjoint, and each expert only rated one object per group)
  3. Give up
  4. Something else (mixed effect model, etc.)?


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.