# Unable to calculate kappa or weighted kappa when 1+ raters give the same rating to everyone -- what is a good solution?

Are there alternate inter-rater reliability statistical tests one can perform in cases where a rater gives the same rating to everyone? SPSS is unable to return a kappa or weighted kappa value when such is the case. Or, for one of my variables, one rater (there are 2 total) gave the same rating to everyone and the second rater gave the same rating for all except 1 subject). SPSS returned k=0, even though the percent agreement is 95% (there are 20 total subjects; 19/20 agreed). How am I to interpret this? Thanks!

This is a well-known "paradox" of chance-adjusted indexes of categorical agreement that use the observed distributions of the categories to estimate chance agreement (e.g., kappa, pi, and alpha). See the papers cited below. In brief, kappa is only high when observed agreement is higher than estimated chance agreement. In cases like yours, kappa is low because both observed agreement and estimated chance agreement are high. To get around this, you can either use an alternative chance-adjusted index of categorical agreement that uses a category-based model of chance-agreement (e.g., the $$S$$ coefficient) or use category-specific agreement (e.g., positive agreement and negative agreement for binary classes).