Linear and Quadratic Weighted Kappa SPSS Error

Question

I am using both Linear and Quadratic weighted kappa to examine the extent to which two measures (a quantitative clinical interview, and a corresponding self-report questionnaire) agree at the item level. By this I mean that I am checking extent to which the score for each item on the clinical interview, matches the score to the corresponding item on the questionnaire. All items on both measures are rated on a frequency likert scale (ranging from 1 "never", to 6 "every day").

However, the SPSS package I am using keeps returning with the following error :

"All ratings are the same for at least one rater. This command is not executed."

I believe this is because in the case of one instrument, the interview has rated all participants as 1/"never".

How do I resolve this issue?

Answer 1

The problem is that where one or both of the raters gives the same rating to everybody you cannot compute $\kappa$. The usual recourse is to report percent agreement and ay why $\kappa$ was not available.

There is a pair of articles by Feinstein, A R and Cicchetti, D V entitled High agreement but low kappa: {I} the problems of two paradoxes and High agreement but low kappa: {II} resolving the paradoxes

There is also an article by Spitznagel, E L and Helzer, J E A proposed solution to the base rate problem in the kappa statistic which may help.

They are about the situation where the base rate (or prevalence) is low whereas in your situation the rate is zero but I think they may help you and also give you an entrance into the literature.