0
$\begingroup$

Consider an experiment where randomly selected experts from a set are asked to attribute Likert-scale ratings to randomly selected objects from two groups A and B. Now, suppose I want to test whether there is a statistically significant difference between the distributions of ratings for objects in groups A and B. Rating distributions are strongly non-normal.

My first thought would have been to use Mann-Whitney U but this experimental design violates multiple independence assumptions. Within a single group, multiple ratings may come from the same expert OR refer to the same object. Between groups, some experts may have rated objects both in group A and B.

What is the statistically correct approach to perform this test? I can think of several ways to proceed:

  1. Use a different approach which does not require independence (if so, which one? Permutation test?)
  2. Perform a Mann-Whitney test on a subset of ratings (e.g., ensure that the set of experts for group A and B is disjoint, and each expert only rated one object per group)
  3. Give up
  4. Something else (mixed effect model, etc.)?
$\endgroup$

0

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.