# Quantifying consistency between two groups of raters

I have two groups of people, group A (62 people) and group B (8 people), and each person in each group responded to a test with the same 8 questions where the answer to each question was one of the numbers 1, 2, 3, 4, or 5 (5-point Likert scale).

I want to be able to express how similar each group was at answering the questions, so that if group A answered the questions very similarly to group B, I can say that group A can be used in place of group B.

Is there a statistical test that applies to this situation? I have been looking at Cronbach's alpha and Cohen's kappa, but I cannot tell if either is really appropriate to the situation.

## 2 Answers

The limitation of Cohen's Kappa is that it is meant to compare two raters with usually two categories. A weighted version of Cohen's Kappa is available that allows you to expand the two category formulation, but the issue of multiple raters is largely unexplored in the statistical world.

The bigger issue is that Cohen's Kappa applies to ratings in which there is some latent notion of a "true rating" which is unobserved by you, the statistician. An example of ratings is the presence/absence of cancer in a mammographic screen by two radiologists, or depressive symptoms in an elderly patient evaluated by two psychologists... basically ratings have their stakes in diagnostic medicine. An entirely different tool is used for "gold standard evaluation" when the actual value of the screen or test is known.

If these Likert scales represent, say, opinions or sentiments as on a questionnaire, you would not consider these "ratings" though that term is consistent with the conventional vernacular. They are simply responses.

Coincidentally, there are other measures of consistency (or internal validity) in survey design that you might consider. The Cronbach's Alpha is a measure of how two people, all things similar, might be expected to report in a similar way on a survey. But this application has the alternate application of survey validation.

Bearing that in mind, perhaps there is something more you can say about what the purpose of the survey is.

• Group B is a group of doctors being asked to rate the quality of 8 images, while group A is a group of non-doctors being asked to rate the quality of the same 8 images. The Likert scale represents the person's opinion of the quality of the image. It is much easier to get access to group A, so the idea is to test whether group A can be used in place of group B for the quality rating task. Jun 30, 2015 at 14:35
• @derNincompoop This is just an ANOVA. Jun 30, 2015 at 17:16
• Would I be correct in saying I need to look at a two-way repeated measures ANOVA? Jun 30, 2015 at 17:27
• @derNincompoop yes Jun 30, 2015 at 19:19
• I have been looking at this for a while and don't quite know how I would set this up. Do I take the question to be one factor, and the group to be the second factor, and the responses are the repeated measures? Jun 30, 2015 at 20:33

So i understand that your data is ordinal, correct? If so, i would suggest for instance the measure mentioned by Marek Walesiak in his paper "distance measure for ordinal data". However, since the formula is too complex to write it down, i think the best is to look it up here.