Would it be appropriate to explain a kappa statistic as a chance-adjusted percentage? The folks ultimately receiving results understand percent agreement more easily, but we do want to use the kappa.

Would a .7 kappa be an adjusted 70% agreement?

I understand the math behind Cohen's kappa, but it's really Fleiss' kappa I'm using more, I think (multiple raters).

  • $\begingroup$ It's the same number. $\endgroup$ – Nick Cox Dec 18 '15 at 0:13
  • $\begingroup$ Fleiss' kappa is also basically (actual agreement minus chance agreement) over (1 - chance agreement), then? And it is then safe to say it's really just an adjusted percent, taking out the by-chance element? $\endgroup$ – ShannonC Dec 18 '15 at 0:27
  • $\begingroup$ I don't see what's troubling you. A choice between proportion and percentage is just one of display format. The important thing is the formula; words that are used are conventional. The fact that kappa can be negative might seem most difficult to explain; with luck you'll never have results that poor. This seems more about your audience or readership than concepts: what can you tell us about that? Adjustment for chance agreement is important whatever you decide. $\endgroup$ – Nick Cox Dec 18 '15 at 0:35
  • $\begingroup$ It really is just a matter of wording for the audience. I was asked if I could translate the kappa into percent agreement, to skip needing to explain too far into what the kappa is, beyond 'it accounts for chance.' So I wanted to make sure that I was verbalizing properly and not missing something. I was thinking, looking at the underlying math, that it is basically saying the percent reliability beyond chance; i.e. .7 kappa is 70% agreement beyond that generated by chance. Wanted confirmation that this wasn't crazy! $\endgroup$ – ShannonC Dec 18 '15 at 0:59
  • $\begingroup$ I'd describe it as a measure of agreement with adjustment for chance agreement. Whether you report with maximum 1or 100% is a separate presentation point. $\endgroup$ – Nick Cox Dec 18 '15 at 1:01

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.