3
$\begingroup$

I have the following result from carrying out Cohen's kappa in R

library(irr)
n = 100
o = c(rep(0,n), rep(1,n))
p = c(rbinom(n,1,0.5), rbinom(n,1,0.51))
k = kappa2(
  data.frame(p,o), "unweighted"
)
k

Which outputs

 Cohen's Kappa for 2 Raters (Weights: unweighted)

 Subjects = 200 
   Raters = 2 
    Kappa = -0.08 

        z = -1.13 
  p-value = 0.258 

My interpretation of this

the test is displaying that there seems to be disagreement between the two vectors as kappa is negative. However, given the p value of 0.258 we can't say that this disagreement is significant, and may just be down to chance.

If someone could highlight if there is anything I'm missing from this interpretation that would be appreciated.

$\endgroup$
  • 2
    $\begingroup$ Please use seeded-random data (set.seed()) so we get a reproducible example. Also, try other package implementations such as DescTools::CohenKappa(), it gives you lower and upper confidence intervals which might be more meaningful to decide whether you can conclude there was no agreement/disagreement. $\endgroup$ – smci Apr 23 at 8:45
4
$\begingroup$

From the perspective of an applied analyst:

First note: that disagreement means if rater A says 1 rater B says 0; it is like how a Pearson correlation of -1 denotes a strong, albeit negative, relationship. The actual null hypothesis here is: what rater A says has no relation to what rater B says.

I wouldn't make such vague yet absolute declarations such as "there seems to be disagreement" (or rather there seems to be no agreement). It is not really an appropriate summary of data without significant background and context. If we had that background and context (such as in a discussions section), we could contribute some nuanced synthesis of the result, pointing to improvements or reasons for disagreement, etc.

To interpret the results:

  • report the percentage agreement, note if any one category was more prevalent (a case when % agreement may be high but $\kappa$ may be low)
  • state the kappa statistic and it's confidence interval
  • I often question the worth of a p-value where the null hypothesis is a stupid case of "no agreement", but you can quote the p-value and say that the data did not provide evidence that the raters agree.
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.