I'm interested in comparing ratings of the same objects that were done by raters from 2 different GROUPS/CLASSES (Experts, and Semi-Experts), so I can decide whether Semi-experts can replace experts in my setting. Though I thought I'd easily find a recommended method for doing this in the literature, after a long search, I'm not sure this is addressed.
The following figure illustrates the data structure:
(*See additional notes at bottom regarding the data structure)
To do the comparison, I believe I need to:
a. See to what extent experts and semi-experts ratings agree with each other (the easy part, but bear with me. there's a question there too); and
b. Check the inter-rater agreement within each group and say if they are distinguishable from each other.
I've searched the literature. Doing (a) seems straightforward with Krippendorf's alpha. My dataset (which is from real data, not a designed experiment) includes multiple ratings (0-3) per object from each raters' group (experts, semi-experts).
I thought of averaging ratings per object, per group, thus creating a dataset with 2 rows that emulate two "raters" (the prototypical expert and the prototypical semi-expert). Then use Krippendorf's alpha. Here's an illustration:
It's not clear to me, however, that this averaging process is a good idea. I could not find a clear answer to that in Krippendorf's content analysis book, or papers.
(I chose Krippendorf's alpha as in some cases I have more than 2 raters; plus it's more robust to missing data and other issues. But Krippendorf designed his alpha to check for consistency in agreement, not to compare raters from different groups; and it seems to be the case with other common measures as well. So I'm hesitant).
As for (b), say I get K-alpha (experts) = x, K-alpha (semi-experts) = y. Is there a way to test the significance of the difference?
I've spent a good few days trying to find an answer, so this is not a simple case. Please bother answering only if you really know this stuff deeply. Many thanks for your consideration!
Notes about the data structure
I believe these two comments should not make a difference for an analysis based on Krippendorf's alpha. But I bring there here for completeness:
- Not all objects have ratings from all raters. Mostly, each object was rated by a couple of raters from each group)
- The number of raters in each group is different (30 in one, 60 in the other).