1
$\begingroup$

I'm trying to determine inter-rater reliability for ratings produced by 4 unique raters, with four possible ratings, NA, 0, 1, 2 corresponding to a quality judgement. I am interested in the distance between ordinal ratings, but also need to take into account the nominal rating NA which are valid responses. Is there any way I can do this with Krippendorff's alpha or some alternative estimate of inter-rater agreement? Should I just treat all responses as nominal? Thank you!

$\endgroup$
1
$\begingroup$

If NA is not really a separate category but just the notation for missing data, you can use a generalized formula for inter-rater agreement that allows for missing data and weighting schemes. Then act like you have three ordinal categories: 0, 1, 2.

If NA is a separate category that actually has meaning, ask whether that category is part of the ordinal scale with the other categories. If it is, then you can just treat it as -1 or 3 or wherever it belongs in the scale. If it isn't, then you could perhaps create a custom weighting scheme that gives ordinal credit to 0, 1, 2 and nominal credit to NA:

$$ \pmatrix{ & \text{NA} & 0 & 1 & 2 \\ \text{NA} & 1.0 & 0.0 & 0.0 & 0.0 \\ 0 & 0.0 & 1.0 & 0.5 & 0.0 \\ 1 & 0.0 & 0.5 & 1.0 & 0.5 \\ 2 & 0.0 & 0.0 & 0.5 & 1.0\\ } $$

$\endgroup$
2
  • $\begingroup$ Hi, thank you for your super helpful response! NA is a separate category and doesn't have a place on the ordinal scale with the other categories. Regarding creating custom weighting schemes for Krippendorff's alpha in R, would you be able to suggest any reading on how to go about doing this (for beginners to inter-rater agreement). Thank you again for your time! $\endgroup$
    – Jen
    May 11 at 7:47
  • $\begingroup$ I put together a little example for you using my agreement R package. rpubs.com/jmgirard/custom_weights $\endgroup$ May 11 at 21:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.