3
$\begingroup$

Cohens kappa can be used as a measure of interrater agreement. However, sometimes the theoretical maximum of kappa < 1 and it may be more correct to calculate kappa as the proportion of the maximum value of kappa.

I need a good calculation example for a 2x2 matrix of how to calculate the maximum value of kappa.

It can be calculated at: http://vassarstats.net/kappa.html, but when I follow the example by Umesh I cannot get the numbers to match with the online calculator.

$\endgroup$
  • $\begingroup$ The largest value in any diagonal cell is the minimum of the respective row and column totals. Does that help clarify it? $\endgroup$ – mdewey Nov 26 '16 at 21:37
1
$\begingroup$

The largest value in any diagonal cell is the minimum of the respective row and column total. Thus you can see what the maximum possible agreement would be which will usually be less than 100%. Only if the margins are the same do you get the possibility of perfect agreement with everything in the diagonal.

$\endgroup$
1
$\begingroup$

By definition, $\kappa = \frac{p_o - p_e}{1 - p_e} = 1- \frac{1 - p_o}{1 - p_e}$, where:

  • $p_e$ is the probability of chance agreement
  • $p_o$ is the observed proportionate agreement

Since $p_e \in [0,1]$, we have $1 - p_e \geq 0$.

Similarly, since $p_o \in [0,1]$, we have $1 - p_o \geq 0$.

Therefore $\frac{1 - p_o}{1 - p_e} \geq 0$

As a result, $\kappa$ is maximized when $\frac{1 - p_o}{1 - p_e} = 0$, i.e. $1 - p_o = 0$, i.e. $p_o=1$, in which case $\kappa=1$.

$p_o=1$ corresponds to the case where annotators never disagreed.


In the example of a 2x2 matrix representing the agreement count between annotator $A$ and annotator $B$:

enter image description here

$\kappa = 1 \Leftrightarrow p_o=1 \Leftrightarrow \frac{a + d} {a+b+c+d} = 1 \Leftrightarrow b = c =0$

$\endgroup$
  • $\begingroup$ Hi Frank. It seems like this is the regular kappa calculation. What I needed as the kappa(max) calculation. If you can update your answer to take into account kappa(max) i shall be happy to upvote your answer. $\endgroup$ – Kasper Christensen Nov 27 '16 at 11:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.