Inter rater reliability with multiple raters, multiple categorical variables with different levels

Question

Apologies if the answer to this question exists elsewhere, but I haven't found anything quite matching what I'm looking for.

I have a dataset wherein 3 raters independently assessed 10 subjects by answering an identical set of 27 questions. Of the questions, 11 have mutually-exclusive binary responses (either "yes" or "no"), and 16 have mutually-exclusive ternary responses ("yes, "no", or "unknown").

I would like to assess inter rater reliability (IRR) between the raters. I would like to know:
a) Is there is a single test I could use to assess IRR across all of this data at once?
b) If not, how is best to subset the data for applying an IRR test to parts of it? (e.g. pairwise comparisons between raters across all questions and research papers?)

I'm working in R, so if you have specific tests to recommend, I would also be grateful if you could point me in the direction of any good packages to perform them.

Thanks in advance for your help!

Answer 1

It is probably best to assess the raters' agreement per question separately. You could use any chance-adjusted index of categorical agreement for both type of question; you'd just have to decide how to treat the "unknown" option (e.g., as a third nominal option, as a third ordinal option between yes and no, or as missing data). If, for some reason, you really need to combine the reliability across the questions, then you can calculate the index per question and then average them.

If all of these questions are meant to be measuring the same exact thing in 27 different ways, then you could look into a generalizability (G) study implemented as a logistic mixed effects model (i.e., nesting responses within raters and questions). But that seems unlikely and probably overkill.

If you are using R, you can look at the following packages: irr, irrCAC, or agreement.