I'm trying to calculate interrater agreement between two judges. For each item, we give them 5 alternatives and ask them to rate 1-7, in respect to their confidence it belongs to a given category. So, for each subject, more than one alternative can have the same rating. This violates the mutual exclusivity principle of Kappa.
I've become aware of the free-marginal multirater kappa (Kfree); However, this is still now what I'm looking for: I want to use a linear weight as a disagreement of 6-7 should not be weighted in the same way as a disagreement between 1-7. Any suggestions? thank you a lot!