# Rotating raters and kappa - appropriate statistic?

I have a several hundred x-rays. Each case is reviewed by 2 of 5 raters and assigned a categorical severity (Mild/Moderate/Severe). So similar data I expect might look like:

+------+--------+--------+--------+
| Case | Rater1 | Rater2 | Rater3 |
+------+--------+--------+--------+
|    1 |      1 |      1 |        |
|    2 |      1 |        |      2 |
|    3 |        |      3 |      3 |
|    4 |      1 |      1 |        |
|    5 |      1 |        |      2 |
|    6 |        |      3 |      3 |
|    7 |      1 |      1 |        |
|    8 |      1 |        |      2 |
|    9 |        |      3 |      3 |
+------+--------+--------+--------+


I'm familiar with the situation where I have multiple raters evaluating ALL cases, but not so familiar with rotating raters like the above. Is Fleiss Kappa still an appropriate statistic to evaluate inter-rater reliability and agreement between the three readers? I imagine I could look at 1 vs 2, 1 vs 3, and 2 vs 3 as individual subsets omitting missing data?

I have yet to commence this study (so the above is example data), would it be more prudent to have some fraction of cases evaluated by all raters? If so, any resources on how to approach determining the appropriate # of cases to assign to all vs just 2 raters? -Thanks!

You have ordinal categories and missing data. I recommend estimating inter-rater reliability using a generalized agreement coefficient that can accommodate these properties. For instance, you could set up your data in an $$n$$-by-$$r$$ matrix where $$n$$ is the number of x-rays and $$r$$ is the number of unique raters. Each cell in this matrix $$(i,j)$$ would contain $$\{1$$, $$2$$, or $$3\}$$ to indicate th rating made on x-ray $$i$$ by rater $$j$$. If x-ray $$i$$ was not rated by rater $$r$$, treat that cell as missing (e.g., NA). Then use a generalized agreement coefficient (as described in the citations below).

For an accessible introduction, see:

Gwet, K. L. (2014). Handbook of inter-rater reliability: The definitive guide to measuring the extent of agreement among raters (4th ed.). Advanced Analytics. https://www.agreestat.com/

For a more recent and technical take, see:

Van Oest, R. & Girard, J. M. (in press). Weighting Schemes and Incomplete Data: A Generalized Bayesian Framework for Chance-Corrected Interrater Agreement. Psychological Methods. https://psyarxiv.com/s5n4e/