4
$\begingroup$

Stats newbie here, please bear with me.

I've been researching inter-rater reliability in order to figure out what I will be using for my thesis study. The most common used test for this in my field/department is Cohen's Kappa. While researching it, I ran into varied papers criticizing it and I am also seeing questions here about how to report on seemingly strange Kappa values.

I ran into a paper written by Gwet in 2002 "Kappa Statistic is not Satisfactory for Assessing the Extent of Agreement Between Raters" and it shed some light as well as introduced a different alternative (AC1).

However, Cohen's Kappa still seems to be much, much more prevalent. Therefore, I am wondering if I am missing something, hence my question.

Are there good reasons to use Cohen's Kappa over Gwet's AC1?

Improve this question
$\endgroup$
4
  • 2
    $\begingroup$ Sinces Cohen's kappa seems to be the standard in your field, just calculate both. If they agree celebrate, if they don't discuss why they don't agree (e.g. list the poor qualities of Cohen's Kappa). $\endgroup$
    – bdeonovic
    Feb 12, 2016 at 18:45
  • $\begingroup$ Yes this is what I would do but I'm wondering why people in my field are not doing this. Is there a relevant use to choosing Kappa over AC1? And if not, is it habit? Resistance? Lack of awareness of the existence of a more reliable value? But this 2nd part of my question belongs to Academia StackExchange, hence why I want to rule out other possibilities first :-) $\endgroup$
    – curious
    Feb 12, 2016 at 21:51
  • 2
    $\begingroup$ I would not use Kappa, as it lumps together sensitivity and specificity. See Feinstein, A. R. and Cicchetti, D. V. (1990). High agreement but low kappa: I. The problems of two paradoxes. Journal of Clinical Epidemiology, 43(6):543–549, and also Cicchetti, D. V. and Feinstein, A. R. (1990). High agreement but low kappa: II. Resolving the paradoxes. Journal of Clinical Epidemiology, 43(6):551–558. The resolution in the second article is to use separate measures of sensitivity (positive agreement) and specificity (negative agreement). $\endgroup$
    – Alexis
    Feb 18, 2016 at 23:02
  • 2
    $\begingroup$ As to why kappa is so frequently used, to be very frank, I think much of what gets published is rote "cargo cult" statistics based on ritualistically waving around of Greek letters, rather than thinking substantively and critically about the meaning of the methods relative to the research. (I hope that comes off more as an exhortation for us to raise our methodological bars, than rank cynicism.) $\endgroup$
    – Alexis
    Feb 18, 2016 at 23:05

1 Answer 1

Reset to default
3
$\begingroup$

I would argue that Cohen's kappa and Gwet's gamma are both problematic approaches to the estimation of inter-rater reliability. Both make a number of assumptions about the behavior of raters that are rarely tenable in practice and produce paradoxical results when these assumptions are violated (Feng, 2013; Zhao, Liu, & Deng, 2012). The popularity of Cohen's kappa stems largely from historical precedent/inertia and the "intuitive appeal" of its logic (i.e., applying Bayes' rule to the estimation of chance agreement). The resistance of the field to change to Gwet's gamma is probably more due to sociological reasons than statistical ones, although I think statistical arguments against Gwet's gamma could be made (see again the cited articles). Unfortunately, a truly effective alternative has not yet been developed. At the moment, your best bet is probably to report multiple measures. As @Alexis stated in her comments, one attractive option is to use specific agreement for each category as suggested by Cicchetti & Feinstein (1990).

References

Cicchetti, D. V, & Feinstein, A. R. (1990). High agreement but low kappa: II. Resolving the paradoxes. Journal of Clinical Epidemiology, 43(6), 551–558.

Feng, G. C. (2013). Factors affecting intercoder reliability: A Monte Carlo experiment. Quality & Quantity, 47(5), 2959–2982.

Zhao, X., Liu, J. S., & Deng, K. (2012). Assumptions behind inter-coder reliability indices. In C. T. Salmon (Ed.), Communication Yearbook (pp. 418–480). Routledge.

Improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.