I have a dataset in which the behaviour of $n = 6$ subjects have been rated by $k = 84$ raters on an ordinal scale with 5 ordinal categories. The distances between the ordinal categories are not equally spaced (and should not be assumed to stem from a continuous scale). However, raters were also instructed to use a NA category, if they felt that the behaviour could not be observed for this particular subject and should therefore not be honoured by a numerical category.

I now want to calculate the inter-rater agreement (and in a later step also the validity of the rating against a "gold standard").

If I only had the 5 numerical, ordinal categories, I would have used Gwet's $AC_2$ with quadratic weights. However, as I now have to integrate the sixth category of NA ("not observable"), I'm not sure, if I can stay with this measure or if I would need to switch to a nominal coefficient (losing much information in the process). The inclusion of the sixth category is mandatory, as in some of the subjects the use of the NA category would be the "gold standard" (i.e. the assignment of a numerical value would actually be "wrong").

If I can stay with Gwet's $AC_2$, where would the NA be "positioned"? At the lower or upper end of the scale (i.e. maximum distance to the highest category or lowest category)?

Bonus: Is there a way to also score such an agreement coefficient against the mentioned gold standard values using R? I have found a formula for Gwet's $AC_1$ in his book (Handbook of Inter-Rater Reliability (3rd Edition)), but I'm not skilled enough to implement the analysis in R. Also, if the coefficient changes based on my first question, this would be not the right way to go.


Usually, missing values are discarded (listwise) when using Cohen's kappa or variants thereof, and this should not have a strong impact on the results (see, e.g., De Raadt et al., Kappa Coefficients for Missing Data, Educational and Psychological Measurement, 79(3), 2019), especially if missing is at random and there's not too much missing data, of course. If the "NA" category needs to be treated alongside other response levels, you can probably define a custom weight matrix for the weighted Cohen’s Kappa. (And if you worry about treating response categories as unordered levels, you can still use Krippendorff’s alpha.) You could also analyze the data in two steps: numerical ratings first, then "NA" ratings, if that make sense of course (I can think of some applications in psychiatric diagnosis, but I don't know about your setup).

Regarding R functions, I you don't want to write your own routine, you can try the packages available at AgreeStat Analytics, or DiagnosisMed if it is still available on CRAN or elsewhere (apparently, this package only offers Gwet's AC1 index).

Edit (in response to comments)

Your idea makes sense, which also means that positioning "NA" at either end of the scale works too, since you can arrange the weight matrix so that agreement is greater for cells nearer the main diagonal using $w_{ab} = 1−|𝑎−𝑏|/(𝐼−1)$, or $w_{ab}=1−(𝑎−𝑏)^2/(𝐼−1)^2$, as suggested by Fleiss & Cohen, The usual caveats of Kappa still hold, though (its value depends strongly on the marginal distributions).

The idea of the two-step approach follows from skip-items in screening tool (e.g., if patient A has compulsive disorder, how would you rate item x or y; otherwise skip to the next section) or tailored checklists where "can't asnwer", "don't understand" or "doesn't apply" response are allowed. Both situations imply that we need to analyse those subsets of items separately. Of course, this may not be applicable in your case, especially if there are a lot of "NA", or if "NA" values are really part of the rating scale.

  • $\begingroup$ Thank you for your input on the issue! For my setup, the NAs must not be discarded but rather treated appropriately. I was thinking about a custom matrix but I wouldn't know, where to position the NA category. Would it make sense to for example weigh the numericals as squared and the NAs without any weight towards the others? Could you maybe specify on how you would approach the two step approach? $\endgroup$ – Dom42 Nov 21 '20 at 15:47
  • $\begingroup$ Once again thanks for your input! As NA really is a valid answer (in a sense that the numerical scale is not applicable to the subject in question), I believe the two step approach would not be fitting. Concerning the positioning of the NA: in my case, assigning a low numerical value instead of a "true" NA is more severe, so in my opinion, positioning NA at the higher end of the scale seems to be more appropriate (i.e. if a rater mistakenly assigns a numerical, it's worse if he assigns a low number then a large one). If this is not wrong from a statistical point of view, I would go with that. $\endgroup$ – Dom42 Nov 21 '20 at 19:06
  • $\begingroup$ That makes sense, and you will probably find a good weight matrix to handle this scenario (and, optionally, convert Kappa to other measure like Gwet's indices). I would, however, recommend to analyze the "NA" response patterns separately, using raw concordance or anything else, in order to check that this extra category is chosen wisely by a majority of raters. $\endgroup$ – chl Nov 21 '20 at 20:17
  • $\begingroup$ Thanks again. I will do so. One more thing concerning the response pattern: as specified in my question, there is a "gold standard" for every subject that was specified by experts. Do you have any ideas/pointers for the inclusion of such a measure in case of ordinal data? Especially with R? I already used the package irrCAC which was mentioned in your original answer to calculate the Gwet $AC$ with a custom modification for random subjects/random raters, but I don't know if there is a better alternative to also include the gold standard in R. $\endgroup$ – Dom42 Nov 22 '20 at 13:07
  • 1
    $\begingroup$ @Dom42 See, e.g., Using Pooled Kappa to Summarize Interrater Agreement across Many Items (which discusses averaged vs. pooled $\kappa$), or Inter-rater reliability of the Extremism Risk Guidelines 22+ for an application with a gold standard. At this point, I believe you should rather ask a new question, with link to this one and maybe the other thread you mentioned above. $\endgroup$ – chl Nov 24 '20 at 18:49

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.