0
$\begingroup$

Are there alternate inter-rater reliability statistical tests one can perform in cases where a rater gives the same rating to everyone? SPSS is unable to return a kappa or weighted kappa value when such is the case. Or, for one of my variables, one rater (there are 2 total) gave the same rating to everyone and the second rater gave the same rating for all except 1 subject). SPSS returned k=0, even though the percent agreement is 95% (there are 20 total subjects; 19/20 agreed). How am I to interpret this? Thanks!

$\endgroup$
0
$\begingroup$

This is a well-known "paradox" of chance-adjusted indexes of categorical agreement that use the observed distributions of the categories to estimate chance agreement (e.g., kappa, pi, and alpha). See the papers cited below. In brief, kappa is only high when observed agreement is higher than estimated chance agreement. In cases like yours, kappa is low because both observed agreement and estimated chance agreement are high. To get around this, you can either use an alternative chance-adjusted index of categorical agreement that uses a category-based model of chance-agreement (e.g., the $S$ coefficient) or use category-specific agreement (e.g., positive agreement and negative agreement for binary classes).

  • Feinstein, A. R., & Cicchetti, D. V. (1990). High agreement but low kappa: I. The problems of two paradoxes. Journal of Clinical Epidemiology, 43(6), 543–549. https://doi.org/10/fwqv5m

  • Cicchetti, D. V., & Feinstein, A. R. (1990). High agreement but low kappa: II. Resolving the paradoxes. Journal of Clinical Epidemiology, 43(6), 551–558. https://doi.org/10/czkxkb

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.