0
$\begingroup$

I'm trying to calculate interrater agreement between two judges. For each item, we give them 5 alternatives and ask them to rate 1-7, in respect to their confidence it belongs to a given category. So, for each subject, more than one alternative can have the same rating. This violates the mutual exclusivity principle of Kappa.

I've become aware of the free-marginal multirater kappa (Kfree); However, this is still now what I'm looking for: I want to use a linear weight as a disagreement of 6-7 should not be weighted in the same way as a disagreement between 1-7. Any suggestions? thank you a lot!

$\endgroup$
0
$\begingroup$

You can calculate a separate kappa score for each alternative, but a single kappa score for all alternatives will not be possible with traditional formulations.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.