I have a dataset in which the behaviour of $n = 6$ subjects have been rated by $k = 84$ raters on an ordinal scale with 5 ordinal categories. The distances between the ordinal categories are not equally spaced (and should not be assumed to stem from a continuous scale).
However, raters were also instructed to use a
NA category, if they felt that the behaviour could not be observed for this particular subject and should therefore not be honoured by a numerical category.
I now want to calculate the inter-rater agreement (and in a later step also the validity of the rating against a "gold standard").
If I only had the 5 numerical, ordinal categories, I would have used Gwet's $AC_2$ with quadratic weights. However, as I now have to integrate the sixth category of
NA ("not observable"), I'm not sure, if I can stay with this measure or if I would need to switch to a nominal coefficient (losing much information in the process). The inclusion of the sixth category is mandatory, as in some of the subjects the use of the
NA category would be the "gold standard" (i.e. the assignment of a numerical value would actually be "wrong").
If I can stay with Gwet's $AC_2$, where would the
NA be "positioned"? At the lower or upper end of the scale (i.e. maximum distance to the highest category or lowest category)?
Bonus: Is there a way to also score such an agreement coefficient against the mentioned gold standard values using
R? I have found a formula for Gwet's $AC_1$ in his book (Handbook of Inter-Rater Reliability (3rd Edition)), but I'm not skilled enough to implement the analysis in
R. Also, if the coefficient changes based on my first question, this would be not the right way to go.