I have $N$ documents. Each document $N_i$ is annotated by a variable number of raters $n_i \in [3, 10]$ into two categories. Is there a way to compute the Kappa statistic for these annotations? My understanding from Wikipedia is that Fleiss' Kappa would not work, as it assumes that all $n_i$ are equal.
$\begingroup$
$\endgroup$
Add a comment
|
1 Answer
$\begingroup$
$\endgroup$
5
You can do this using a "generalized formula." The intuition here is to use the following formula for observed agreement, which uses the number of raters rather than conditionals for specific raters.
$$p_o=\frac{1}{n'}\sum_{i=1}^{n'}\sum_{k=1}^q\frac{r_{ik}(r_{ik}^\star-1)}{r_i(r_i-1)}$$
where $n'$ is the number of items that were coded by two or more raters,
$q$ is the number of possible categories,
$r_{ik}$ is the number of raters that assigned item $i$ to category $k$,
and $r_i$ is the number of raters that assigned item $i$ to any category.
$r_{ik}^\star$ allows you to use a weighting scheme to account for non-nominal categories if desired.
Click the links below to get more information from my website or read Gwet (2014).
Formula and MATLAB function for generalized Cohen's kappa
Formula and MATLAB function for generalized Scott's pi (AKA Fleiss' kappa)
Reference
Gwet, K. L. (2014). Handbook of inter-rater reliability: The definitive guide to measuring the extent of agreement among raters (4th ed.). Gaithersburg, MD: Advanced Analytics.
Uebersax, J. S. (1982). A design-independent method for measuring the reliability of psychiatric diagnosis. Journal of Psychiatric Research, 17(4), 335–342.
answered Jan 3, 2017 at 0:03
-
1$\begingroup$ Thanks for the answer and the links! Was this method invented by Uebersax or Gwet? $\endgroup$– mossaabCommented Jan 3, 2017 at 0:48
-
4$\begingroup$ Interesting question. The advances came in iterations. Fleiss (1971) allows multiple raters but requires the number of raters to be constant. Fleiss and Cuzick (1979) allows multiple and variable raters, but only for two categories. Uebersax (1982) allows for multiple and variable raters and multiple categories but only for nominal categories. Finally, Gwet (2014) or some earlier edition of the same book allows for multiple and variable raters, multiple categories, and weighting schemes. $\endgroup$ Commented Jan 3, 2017 at 0:57
-
$\begingroup$ Another possibility is krippendorffs alpha. $\endgroup$ Commented Jun 13, 2017 at 20:50
-
$\begingroup$ Nice answer @JeffreyGirard, thank you! Any tips on computing inferential statistics? Wondering if there's a closed form, approximation, or if one should just bootstrap {rater, item}s. $\endgroup$– kholCommented Dec 10, 2022 at 6:32
-
$\begingroup$ @khol Gwet provides normal approximations. I usually just bootstrap. $\endgroup$ Commented Jan 3, 2023 at 21:34