Quantifying consistency between two groups of raters I have two groups of people, group A (62 people) and group B (8 people), and each person in each group responded to a test with the same 8 questions where the answer to each question was one of the numbers 1, 2, 3, 4, or 5 (5-point Likert scale).
I want to be able to express how similar each group was at answering the questions, so that if group A answered the questions very similarly to group B, I can say that group A can be used in place of group B.
Is there a statistical test that applies to this situation? I have been looking at Cronbach's alpha and Cohen's kappa, but I cannot tell if either is really appropriate to the situation.
 A: The limitation of Cohen's Kappa is that it is meant to compare two raters with usually two categories. A weighted version of Cohen's Kappa is available that allows you to expand the two category formulation, but the issue of multiple raters is largely unexplored in the statistical world. 
The bigger issue is that Cohen's Kappa applies to ratings in which there is some latent notion of a "true rating" which is unobserved by you, the statistician. An example of ratings is the presence/absence of cancer in a mammographic screen by two radiologists, or depressive symptoms in an elderly patient evaluated by two psychologists... basically ratings have their stakes in diagnostic medicine. An entirely different tool is used for "gold standard evaluation" when the actual value of the screen or test is known.
If these Likert scales represent, say, opinions or sentiments as on a questionnaire, you would not consider these "ratings" though that term is consistent with the conventional vernacular. They are simply responses. 
Coincidentally, there are other measures of consistency (or internal validity) in survey design that you might consider. The Cronbach's Alpha is a measure of how two people, all things similar, might be expected to report in a similar way on a survey. But this application has the alternate application of survey validation.
Bearing that in mind, perhaps there is something more you can say about what the purpose of the survey is.
A: So i understand that your data is ordinal, correct? If so, i would suggest for instance the measure mentioned by Marek  Walesiak in his paper "distance measure for ordinal data". However, since the formula is too complex to write it down, i think the best is to look it up here.
