I am using both Linear and Quadratic weighted kappa to examine the extent to which two measures (a quantitative clinical interview, and a corresponding self-report questionnaire) agree at the item level. By this I mean that I am checking extent to which the score for each item on the clinical interview, matches the score to the corresponding item on the questionnaire. All items on both measures are rated on a frequency likert scale (ranging from 1 "never", to 6 "every day").

However, the SPSS package I am using keeps returning with the following error :

"All ratings are the same for at least one rater. This command is not executed."

I believe this is because in the case of one instrument, the interview has rated all participants as 1/"never".

How do I resolve this issue?

  • $\begingroup$ The short answer is that you cannot calculate agreement here. Just report what happened in your report. It is quite common. $\endgroup$ – mdewey Aug 10 '17 at 15:03
  • $\begingroup$ Thank you mdewey, do you have a reference or any resources where I read further about this limitation? $\endgroup$ – Alice Phillips Aug 17 '17 at 10:37

The problem is that where one or both of the raters gives the same rating to everybody you cannot compute $\kappa$. The usual recourse is to report percent agreement and ay why $\kappa$ was not available.

There is a pair of articles by Feinstein, A R and Cicchetti, D V entitled High agreement but low kappa: {I} the problems of two paradoxes and High agreement but low kappa: {II} resolving the paradoxes

There is also an article by Spitznagel, E L and Helzer, J E A proposed solution to the base rate problem in the kappa statistic which may help.

They are about the situation where the base rate (or prevalence) is low whereas in your situation the rate is zero but I think they may help you and also give you an entrance into the literature.

| cite | improve this answer | |

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.