Are there any limitations for using Cohen's kappa with sparse data? I need inter-rater agreement between 2 raters for ~15 items, and the data in the contingency table is quite sparse (0 in some cells, lots of values < 5).
$\begingroup$
$\endgroup$
-
$\begingroup$ Are you talking about the effect of zero cells on the magnitude of the coefficient or on the test of significance? We know that the max possible value of kappa is very distribution-dependent - see wikipedia and this. So there is no limitations, but issues of interpretation of magnitude. As for significance testing, zero cells / small sample is a problem; however, you can compute the exact p-value instead of the asymptotic-based. $\endgroup$ – ttnphns Sep 7 '13 at 11:47
$\begingroup$
$\endgroup$
Cohens Kappa is known to have limitations for skewed datasets.
Quoting an example from here
Consider following matrix :
+---+----+
| 1 | 6 |
+---+----+
| 9 | 84 |
+---+----+
The above example has an observed agreement of 0.85 but the Cohen' Kappa is 0.04.
The solution suggested in this article is to report two separate agreement metrics for positive and negative classes.
