How to evaluate differences of multiple Likert scale ratings of multiple items within groups? I have data where different reviewers rated how important multiple topics are on a scale of 1 to 5 (Likert scale). Each of the topics is assigned to a group. I want to see if there any differences in the overall rating of importance between the groups of topics.
Here is an example of what the data looks like...
\begin{array}{|c|c||c|} topic & group & reviewer1 & reviewer2 & reviewer3 \\
    \hline
    1&A&4&2&2\\
    2&A&5&5&5\\
    3&B&3&3&2\\
    4&B&4&5&2\\
    5&C&5&4&4
    \end{array}
The real data has many more topics, groups and reviewers.
I know I shouldn't be using ANOVA so I assume that I should use a nonparametric technique but I'm not sure which one considering there are multiple topics and multiple reviewers in each group. Some reviewers might be more strict than others and I want to account for that. Thanks for your help!
 A: I would use a fixed effects ordered logit regression. If you have many observations per group, just throw in a dummy per group, and then a Wald test tells you whether, say, group A or group B receives significantly different ratings. I added the condition "many observations per group", because estimating group fixed effects via dummies is only valid asymptotically, as illustrated in this post. According to their simulations, you should have about 50 observations per group. 
Since you expect differences by reviewer, which is reasonable, you can also use reviewer fixed effects to account for time-invariant reviewer heterogeneity. This controls for the fact that particularly "grumpy" reviewers may review one group in particular, and without accounting for this, it would look as if that group is less important than it actually is. Adding reviewer fixed effects via dummy, however, makes the above problem more severe, because then you have more parameters to estimate with the same number of observations. Solutions on how to implement fixed effects ordered logit without relying on asymptotics are sketched here; apparently, Stata has not implemented it officially.
You can avoid the above incidental parameters problem with ordered logit if you are willing to assume your Likert scale is not merely ordinal, but cardinal. (Which implies, for example, that on average a 1 and 5 rating is as good as a 2 and 4 rating.) In this case you can simply use OLS fixed effects regression, again using fixed effects for reviewer and group. Testing the group dummies for differences will tell you whether some group is viewed as more important than another.
