3
$\begingroup$

Stats newbie here, please bear with me.

I've been researching inter-rater reliability in order to figure out what I will be using for my thesis study. The most common used test for this in my field/department is Cohen's Kappa. While researching it, I ran into varied papers criticizing it and I am also seeing questions here about how to report on seemingly strange Kappa values.

I ran into a paper written by Gwet in 2002 "Kappa Statistic is not Satisfactory for Assessing the Extent of Agreement Between Raters" and it shed some light as well as introduced a different alternative (AC1).

However, Cohen's Kappa still seems to be much, much more prevalent. Therefore, I am wondering if I am missing something, hence my question.

Are there good reasons to use Cohen's Kappa over Gwet's AC1?

$\endgroup$
  • 2
    $\begingroup$ Sinces Cohen's kappa seems to be the standard in your field, just calculate both. If they agree celebrate, if they don't discuss why they don't agree (e.g. list the poor qualities of Cohen's Kappa). $\endgroup$ – bdeonovic Feb 12 '16 at 18:45
  • $\begingroup$ Yes this is what I would do but I'm wondering why people in my field are not doing this. Is there a relevant use to choosing Kappa over AC1? And if not, is it habit? Resistance? Lack of awareness of the existence of a more reliable value? But this 2nd part of my question belongs to Academia StackExchange, hence why I want to rule out other possibilities first :-) $\endgroup$ – Emilie Feb 12 '16 at 21:51
  • 1
    $\begingroup$ I would not use Kappa, as it lumps together sensitivity and specificity. See Feinstein, A. R. and Cicchetti, D. V. (1990). High agreement but low kappa: I. The problems of two paradoxes. Journal of Clinical Epidemiology, 43(6):543–549, and also Cicchetti, D. V. and Feinstein, A. R. (1990). High agreement but low kappa: II. Resolving the paradoxes. Journal of Clinical Epidemiology, 43(6):551–558. The resolution in the second article is to use separate measures of sensitivity (positive agreement) and specificity (negative agreement). $\endgroup$ – Alexis Feb 18 '16 at 23:02
  • 1
    $\begingroup$ As to why kappa is so frequently used, to be very frank, I think much of what gets published is rote "cargo cult" statistics based on ritualistically waving around of Greek letters, rather than thinking substantively and critically about the meaning of the methods relative to the research. (I hope that comes off more as an exhortation for us to raise our methodological bars, than rank cynicism.) $\endgroup$ – Alexis Feb 18 '16 at 23:05
  • $\begingroup$ Thank you for being frank Alexis. This is helpful as a student who is trying to understand how things work but who got somewhat brushed off because well - "why don't you just do what we normally do". It's really easy to doubt yourself when beginning in stats and faced with people who deal with it on an almost daily basis... I had found the 1990 article this week and figured this could be a way of resolving my problem without making too many waves among my faculty :-) $\endgroup$ – Emilie Feb 19 '16 at 0:23
2
$\begingroup$

I would argue that Cohen's kappa and Gwet's gamma are both problematic approaches to the estimation of inter-rater reliability. Both make a number of assumptions about the behavior of raters that are rarely tenable in practice and produce paradoxical results when these assumptions are violated (Feng, 2013; Zhao, Liu, & Deng, 2012). The popularity of Cohen's kappa stems largely from historical precedent/inertia and the "intuitive appeal" of its logic (i.e., applying Bayes' rule to the estimation of chance agreement). The resistance of the field to change to Gwet's gamma is probably more due to sociological reasons than statistical ones, although I think statistical arguments against Gwet's gamma could be made (see again the cited articles). Unfortunately, a truly effective alternative has not yet been developed. At the moment, your best bet is probably to report multiple measures. As @Alexis stated in her comments, one attractive option is to use specific agreement for each category as suggested by Cicchetti & Feinstein (1990).

References

Cicchetti, D. V, & Feinstein, A. R. (1990). High agreement but low kappa: II. Resolving the paradoxes. Journal of Clinical Epidemiology, 43(6), 551–558.

Feng, G. C. (2013). Factors affecting intercoder reliability: A Monte Carlo experiment. Quality & Quantity, 47(5), 2959–2982.

Zhao, X., Liu, J. S., & Deng, K. (2012). Assumptions behind inter-coder reliability indices. In C. T. Salmon (Ed.), Communication Yearbook (pp. 418–480). Routledge.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.