Macro and Micro Inter-Coder/Rater Agreement

Question

I have a labeling task with 4 labels for which I'm using 2 coders/annotators/raters. This labeling effort leads to a confusion matrix like the one in

https://docs.google.com/drawings/d/1T7pkLLkE7qvKanxqvZO_uQ_q3hz6S3fcTbOnWophx38/edit?usp=sharing

I want to measure the agreement between the 2 annotators. How so? Kappa Coefficient works for binary labeling/classification but not for this case.

I thought of having like a micro kappa for each 2 labels and then an averaged macro Kappa across all combinations. How dummy does that sound?

Thanks

Answer 1

Actually, Cohen's kappa statistic can work just fine for the four labels. The problem may be that you are estimating kappa as an exact agreement, which may not be what you want to measure. Typically, kappa for a scale will be estimated not as exact agreement, but as a weighted agreement. In this case the further from the diagonal the rater scores, the less agreement is credited.

One of the great features of kappa is that it will adjust scales for those sections of the scale that are not use. For example, if a 5-point scale only gets selections of the first three points, then the expected agreement will be based on the frequency of selection of the points on the scale and not just 20%. Kappa is far better than just percent agreement.