Calculate interrater agreement (e.g. Kappa) with linear weighting but when subject can repeat values?

Question

I'm trying to calculate interrater agreement between two judges. For each item, we give them 5 alternatives and ask them to rate 1-7, in respect to their confidence it belongs to a given category. So, for each subject, more than one alternative can have the same rating. This violates the mutual exclusivity principle of Kappa.

I've become aware of the free-marginal multirater kappa (Kfree); However, this is still now what I'm looking for: I want to use a linear weight as a disagreement of 6-7 should not be weighted in the same way as a disagreement between 1-7. Any suggestions? thank you a lot!

Answer 1

You can calculate a separate kappa score for each alternative, but a single kappa score for all alternatives will not be possible with traditional formulations.