Compute inter-rater reliability for categorical ratings with variable raters per target

Question

What is the correct statistic to use for computing inter-rater agreement from data of the following form?

data = pd.DataFrame({
   "rater" : [1, 1, 1, 2, 3, 2, 1, 2, 1, 3],
   "target" : ["a", "b", "c", "a", "a", "c", "d", "b", "f", "g"],
   "rating" : [1, 1, -1, 0, 0, 1, 1, 1, 1, -1]
})

I've been advised to try Cohen's Kappa, but it seems this requires that each target have the same number of raters. In this case, the number of raters per target can be one or more.

I recognize one solution is ICC, but implementations of ICC (eg. pingouin.intraclass_corr) only provides coefficients indicating the reliability of individual raters or the average of raters (Shrout and Fleiss, 1979). But since the the data are nominal I want to take the mode when reviewers disagree, not the mean.

Note that these are toy data. The actual data have 241 unique targets and between 1 and 8 raters (only rating options are -1, 1, and 0).

Thanks in advance!

Answer 1

You can use a generalized formulation of chance-adjusted categorical agreement that allows for a variable number of raters per item (or "target"), but note that items with only a single rater won't contribute to the estimation of agreement.

For example, if you want to use the generalized version of Scott's $\pi$ (and Fleiss' kappa), use:

$$ r_{ik}^\star=\sum_{l=1}^{q}w_{kl}r_{il} $$

$$ p_o=\frac{1}{n'}\sum_{i=1}^{n'}\sum_{k=1}^q \frac{r_{ik}(r_{ik}^\star-1)}{r_i(r_i-1)} $$

$$ \pi_k=\frac{1}{n}\sum_{i=1}^n\frac{r_{ik}}{r_i} $$

$$ p_c=\sum_{k,l}^q w_{kl}\pi_{k}\pi_{l} $$

$$ \pi = \frac{p_o-p_c}{1-p_c} $$

where

$q$ is the total number of categories

$w_{kl}$ is the weight associated with two raters assigning an item to categories $k$ and $l$ (allows you to treat your categories as ordinal or nominal)

$r_{il}$ is the number of raters that assigned item $i$ to category $l$

$n'$ is the number of items that were coded by two or more raters

$r_{ik}$ is the number of raters that assigned item $i$ to category $k$

$r_i$ is the number of raters that assigned item $i$ to any category

$n$ is the total number of items

---

Textbook:

• Gwet, K. L. (2014). Handbook of inter-rater reliability: The definitive guide to measuring the extent of agreement among raters (4th ed.). Gaithersburg, MD: Advanced Analytics.

Software:

• MATLAB = https://mreliability.jmgirard.com
• R = https://www.github.com/jmgirard/agreement
• R = https://cran.r-project.org/package=irrCAC
• (Sorry, I don't use Python)