# Inclusion of a NA category in inter-rater agreement calculation

I have a dataset in which the behaviour of $$n = 6$$ subjects have been rated by $$k = 84$$ raters on an ordinal scale with 5 ordinal categories. The distances between the ordinal categories are not equally spaced (and should not be assumed to stem from a continuous scale). However, raters were also instructed to use a NA category, if they felt that the behaviour could not be observed for this particular subject and should therefore not be honoured by a numerical category.

I now want to calculate the inter-rater agreement (and in a later step also the validity of the rating against a "gold standard").

If I only had the 5 numerical, ordinal categories, I would have used Gwet's $$AC_2$$ with quadratic weights. However, as I now have to integrate the sixth category of NA ("not observable"), I'm not sure, if I can stay with this measure or if I would need to switch to a nominal coefficient (losing much information in the process). The inclusion of the sixth category is mandatory, as in some of the subjects the use of the NA category would be the "gold standard" (i.e. the assignment of a numerical value would actually be "wrong").

If I can stay with Gwet's $$AC_2$$, where would the NA be "positioned"? At the lower or upper end of the scale (i.e. maximum distance to the highest category or lowest category)?

Bonus: Is there a way to also score such an agreement coefficient against the mentioned gold standard values using R? I have found a formula for Gwet's $$AC_1$$ in his book (Handbook of Inter-Rater Reliability (3rd Edition)), but I'm not skilled enough to implement the analysis in R. Also, if the coefficient changes based on my first question, this would be not the right way to go.

Usually, missing values are discarded (listwise) when using Cohen's kappa or variants thereof, and this should not have a strong impact on the results (see, e.g., De Raadt et al., Kappa Coefficients for Missing Data, Educational and Psychological Measurement, 79(3), 2019), especially if missing is at random and there's not too much missing data, of course. If the "NA" category needs to be treated alongside other response levels, you can probably define a custom weight matrix for the weighted Cohen’s Kappa. (And if you worry about treating response categories as unordered levels, you can still use Krippendorff’s alpha.) You could also analyze the data in two steps: numerical ratings first, then "NA" ratings, if that make sense of course (I can think of some applications in psychiatric diagnosis, but I don't know about your setup).

Regarding R functions, I you don't want to write your own routine, you can try the packages available at AgreeStat Analytics, or DiagnosisMed if it is still available on CRAN or elsewhere (apparently, this package only offers Gwet's AC1 index).

Your idea makes sense, which also means that positioning "NA" at either end of the scale works too, since you can arrange the weight matrix so that agreement is greater for cells nearer the main diagonal using $$w_{ab} = 1−|𝑎−𝑏|/(𝐼−1)$$, or $$w_{ab}=1−(𝑎−𝑏)^2/(𝐼−1)^2$$, as suggested by Fleiss & Cohen, The usual caveats of Kappa still hold, though (its value depends strongly on the marginal distributions).
• Thanks again. I will do so. One more thing concerning the response pattern: as specified in my question, there is a "gold standard" for every subject that was specified by experts. Do you have any ideas/pointers for the inclusion of such a measure in case of ordinal data? Especially with R? I already used the package irrCAC which was mentioned in your original answer to calculate the Gwet $AC$ with a custom modification for random subjects/random raters, but I don't know if there is a better alternative to also include the gold standard in R. – Dom42 Nov 22 '20 at 13:07
• @Dom42 See, e.g., Using Pooled Kappa to Summarize Interrater Agreement across Many Items (which discusses averaged vs. pooled $\kappa$), or Inter-rater reliability of the Extremism Risk Guidelines 22+ for an application with a gold standard. At this point, I believe you should rather ask a new question, with link to this one and maybe the other thread you mentioned above. – chl Nov 24 '20 at 18:49