2
$\begingroup$

Are there any limitations for using Cohen's kappa with sparse data? I need inter-rater agreement between 2 raters for ~15 items, and the data in the contingency table is quite sparse (0 in some cells, lots of values < 5).

$\endgroup$
  • $\begingroup$ Are you talking about the effect of zero cells on the magnitude of the coefficient or on the test of significance? We know that the max possible value of kappa is very distribution-dependent - see wikipedia and this. So there is no limitations, but issues of interpretation of magnitude. As for significance testing, zero cells / small sample is a problem; however, you can compute the exact p-value instead of the asymptotic-based. $\endgroup$ – ttnphns Sep 7 '13 at 11:47
2
$\begingroup$

Cohens Kappa is known to have limitations for skewed datasets.

Quoting an example from here

Consider following matrix :

+---+----+
| 1 | 6  |
+---+----+
| 9 | 84 |
+---+----+

The above example has an observed agreement of 0.85 but the Cohen' Kappa is 0.04.

The solution suggested in this article is to report two separate agreement metrics for positive and negative classes.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.