inter-rater agreement for data with nominal and ordinal responses

Question

I'm trying to determine inter-rater reliability for ratings produced by 4 unique raters, with four possible ratings, NA, 0, 1, 2 corresponding to a quality judgement. I am interested in the distance between ordinal ratings, but also need to take into account the nominal rating NA which are valid responses. Is there any way I can do this with Krippendorff's alpha or some alternative estimate of inter-rater agreement? Should I just treat all responses as nominal? Thank you!

Answer 1

If NA is not really a separate category but just the notation for missing data, you can use a generalized formula for inter-rater agreement that allows for missing data and weighting schemes. Then act like you have three ordinal categories: 0, 1, 2.

If NA is a separate category that actually has meaning, ask whether that category is part of the ordinal scale with the other categories. If it is, then you can just treat it as -1 or 3 or wherever it belongs in the scale. If it isn't, then you could perhaps create a custom weighting scheme that gives ordinal credit to 0, 1, 2 and nominal credit to NA:

$$ \pmatrix{ & \text{NA} & 0 & 1 & 2 \\ \text{NA} & 1.0 & 0.0 & 0.0 & 0.0 \\ 0 & 0.0 & 1.0 & 0.5 & 0.0 \\ 1 & 0.0 & 0.5 & 1.0 & 0.5 \\ 2 & 0.0 & 0.0 & 0.5 & 1.0\\ } $$