0
$\begingroup$

I was wondering if there is any particular reason why Cohen's Kappa is defined as this particular ration $\frac{p - r}{1 - r}$, where $p$ is the agreement rate between two, say, classifiers, and $r$ is the rate of agreeing at random.

Why is Kappa not defined, for instance, simply as $\frac{p}{r}$? Is the present form only for the upper limit for Kappa to be $1$?

$\endgroup$
1
$\begingroup$

$$ \kappa = \frac{p_o - p_c}{1 - p_c} $$

This formula does have a deeper meaning and also restricts the range between -1 and 1 (not 0 and 1). Kappa is meant to be the ratio of observed non-chance agreement to possible non-chance agreement.

The numerator ($p_o - p_c$) is the observed non-chance agreement because $p_o$ is the total observed agreement and we are removing the estimated chance agreement $p_c$.

The denominator ($1 - p_c$) is the possible non-chance agreement because $1$ is the maximum value that $p_o$ could take on and we are removing the estimated chance agreement $p_c$.

I think that the ratio of observed agreement to chance agreement ($p_o/p_c$) would be reasonable; it just wouldn't be bounded.

$\endgroup$
0
$\begingroup$

Cohen Kappa is defined so $\kappa$=0 means a random agreement, and $\kappa$=1 is the maximum possible Kappa, that is 100% accuracy.

As far as I know there is no deeper principle behind the formula - just the requirement described (0 -> random, 1 -> maximum). The wikipedia page onb Cohen's kappa list a few other inter-rater agreement measures and links to some papers that describe problems with the measure.

$\endgroup$
  • $\begingroup$ Is it just for easy interpretation? $\endgroup$ – inzl Jan 12 '16 at 14:35
  • $\begingroup$ I believe that is correct. The formula forces the value to be between 0 and 1. $\endgroup$ – William Chiu Jan 12 '16 at 16:19
  • $\begingroup$ I would not put as " just for easy interpretation". Cohen wanted a number with the properties above and thus the formula - so the formula is that because Cohen wanted a number that could be interpreted as 0 -> random 1-> maximum. $\endgroup$ – Jacques Wainer Jan 12 '16 at 20:54

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.