Inter-rater agreement for non-fully crossed designs

Question

I have a set of n subjects to be rated by i raters on a categorical scale that distinguishes k classes. Both the subjects and the raters are randomly sampled from a larger population of subject and raters. Each subject is rated by exactly m raters; however, different raters may rate different subjects. Thus, I have a non-fully crossed design where all subjects are rated by a fixed number of m multiple raters. Dummy data is shown in the image below; the uppercase letters in the cells are the categorical ratings assigned by raters to subjects.

I want to calculate inter-rater agreement; the overall aim of the study is to validate the rating system (the categorical scale of size k), i.e. assess its generalisability to other subjects and rathers from the same populations.

Now the question is: Which inter-rater agreement measure is the most appropriate one? Intraclass correlaction (ICC) is not an option because it is not suitable for categorical data. Cohen's kappa is also not an option, because it is only suitable for fully-crossed designs with exactly two coders, i.e. only if all raters rate all the subjects (see https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3402032/#!po=50.0000).

The most viable candidate seems to be Fleiss' kappa, because it is suitable for studies where any constant number of m coders is randomly sampled from a larger population of coders, with each subject rated by a different sample of m coders (see https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3402032/#!po=50.0000). But is it suitable given that not only the raters but also the subjects are sampled randomly? Besides that, what if I choose to use only two raters (m = 2), can Fleiss' kappa still be applied? There is conflicting information as to the minimum number of raters for Fleiss' kappa, e.g. "three or more raters" (https://www.statisticshowto.datasciencecentral.com/fleiss-kappa) vs. "any number of raters" (https://en.wikipedia.org/wiki/Fleiss%27_kappa).

Besides that, Krippendorff's alpha seems to be an alternative because it generalizes several inter-rater agreement statistics. However, I have also read that it is "more suitable when problems are posed by missing data in fully crossed designs" (https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3402032/#!po=50.0000), so I am not sure if I can use it in the non-fully crossed design.

Answer 1

You can use generalized or "design-independent" formulas for chance-adjusted agreement. These will accommodate the fact that you have a non-crossed design. You can calculate these in R by replacing your empty cells with NA and providing the matrix to the cat_adjusted() function in my agreement package.

• Uebersax, J. S. (1982). A design-independent method for measuring the reliability of psychiatric diagnosis. Journal of Psychiatric Research, 17(4), 335–342. https://doi.org/10/fbbdfn

• Gwet, K. L. (2014). Handbook of inter-rater reliability: The definitive guide to measuring the extent of agreement among raters (4th ed.). Gaithersburg, MD: Advanced Analytics.