2
$\begingroup$

I have $N$ documents. Each document $N_i$ is annotated by a variable number of raters $n_i \in [3, 10]$ into two categories. Is there a way to compute the Kappa statistic for these annotations? My understanding from Wikipedia is that Fleiss' Kappa would not work, as it assumes that all $n_i$ are equal.

$\endgroup$
4
$\begingroup$

You can do this using a "generalized formula." The intuition here is to use the following formula for observed agreement, which uses the number of raters rather than conditionals for specific raters.

$$p_o=\frac{1}{n'}\sum_{i=1}^{n'}\sum_{k=1}^q\frac{r_{ik}(r_{ik}^\star-1)}{r_i(r_i-1)}$$

where $n'$ is the number of items that were coded by two or more raters,

$q$ is the number of possible categories,

$r_{ik}$ is the number of raters that assigned item $i$ to category $k$,

and $r_i$ is the number of raters that assigned item $i$ to any category.

$r_{ik}^\star$ allows you to use a weighting scheme to account for non-nominal categories if desired.

Click the links below to get more information from my website or read Gwet (2014).

Formula and MATLAB function for generalized Cohen's kappa

Formula and MATLAB function for generalized Scott's pi (AKA Fleiss' kappa)

Reference

Gwet, K. L. (2014). Handbook of inter-rater reliability: The definitive guide to measuring the extent of agreement among raters (4th ed.). Gaithersburg, MD: Advanced Analytics.

Uebersax, J. S. (1982). A design-independent method for measuring the reliability of psychiatric diagnosis. Journal of Psychiatric Research, 17(4), 335–342.

$\endgroup$
  • 1
    $\begingroup$ Thanks for the answer and the links! Was this method invented by Uebersax or Gwet? $\endgroup$ – mossaab Jan 3 '17 at 0:48
  • 2
    $\begingroup$ Interesting question. The advances came in iterations. Fleiss (1971) allows multiple raters but requires the number of raters to be constant. Fleiss and Cuzick (1979) allows multiple and variable raters, but only for two categories. Uebersax (1982) allows for multiple and variable raters and multiple categories but only for nominal categories. Finally, Gwet (2014) or some earlier edition of the same book allows for multiple and variable raters, multiple categories, and weighting schemes. $\endgroup$ – Jeffrey Girard Jan 3 '17 at 0:57
  • $\begingroup$ Another possibility is krippendorffs alpha. $\endgroup$ – kjetil b halvorsen Jun 13 '17 at 20:50

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.