Inclusion of a NA category in inter-rater agreement calculation

Question

I have a dataset in which the behaviour of $n = 6$ subjects have been rated by $k = 84$ raters on an ordinal scale with 5 ordinal categories. The distances between the ordinal categories are not equally spaced (and should not be assumed to stem from a continuous scale). However, raters were also instructed to use a NA category, if they felt that the behaviour could not be observed for this particular subject and should therefore not be honoured by a numerical category.

I now want to calculate the inter-rater agreement (and in a later step also the validity of the rating against a "gold standard").

If I only had the 5 numerical, ordinal categories, I would have used Gwet's $AC_2$ with quadratic weights. However, as I now have to integrate the sixth category of NA ("not observable"), I'm not sure, if I can stay with this measure or if I would need to switch to a nominal coefficient (losing much information in the process). The inclusion of the sixth category is mandatory, as in some of the subjects the use of the NA category would be the "gold standard" (i.e. the assignment of a numerical value would actually be "wrong").

If I can stay with Gwet's $AC_2$, where would the NA be "positioned"? At the lower or upper end of the scale (i.e. maximum distance to the highest category or lowest category)?

Bonus: Is there a way to also score such an agreement coefficient against the mentioned gold standard values using R? I have found a formula for Gwet's $AC_1$ in his book (Handbook of Inter-Rater Reliability (3rd Edition)), but I'm not skilled enough to implement the analysis in R. Also, if the coefficient changes based on my first question, this would be not the right way to go.

Answer 1

Usually, missing values are discarded (listwise) when using Cohen's kappa or variants thereof, and this should not have a strong impact on the results (see, e.g., De Raadt et al., Kappa Coefficients for Missing Data, Educational and Psychological Measurement, 79(3), 2019), especially if missing is at random and there's not too much missing data, of course. If the "NA" category needs to be treated alongside other response levels, you can probably define a custom weight matrix for the weighted Cohen’s Kappa. (And if you worry about treating response categories as unordered levels, you can still use Krippendorff’s alpha.) You could also analyze the data in two steps: numerical ratings first, then "NA" ratings, if that make sense of course (I can think of some applications in psychiatric diagnosis, but I don't know about your setup).

Regarding R functions, I you don't want to write your own routine, you can try the packages available at AgreeStat Analytics, or DiagnosisMed if it is still available on CRAN or elsewhere (apparently, this package only offers Gwet's AC1 index).

---

Edit (in response to comments)

Your idea makes sense, which also means that positioning "NA" at either end of the scale works too, since you can arrange the weight matrix so that agreement is greater for cells nearer the main diagonal using $w_{ab} = 1−|𝑎−𝑏|/(𝐼−1)$, or $w_{ab}=1−(𝑎−𝑏)^2/(𝐼−1)^2$, as suggested by Fleiss & Cohen, The usual caveats of Kappa still hold, though (its value depends strongly on the marginal distributions).

The idea of the two-step approach follows from skip-items in screening tool (e.g., if patient A has compulsive disorder, how would you rate item x or y; otherwise skip to the next section) or tailored checklists where "can't asnwer", "don't understand" or "doesn't apply" response are allowed. Both situations imply that we need to analyse those subsets of items separately. Of course, this may not be applicable in your case, especially if there are a lot of "NA", or if "NA" values are really part of the rating scale.