I am working with ordinal scales for cerebral atrophy (ratings 0 to 4) and vascular burden (ratings 0 to 3) assessed by two raters. I have installed the extension bundle for weighted kappa from SPSS (version 25) but I keep getting the error message "some ratings are less than one this command is not executed". I have looked through the threads here and the Internet but as far as I can see nobody has come up with this problem. I know from the literature that intra- and inter-rater reliability are commonly calculated for these scales using weighted kappa. Also, weighted kappa seems to be applicable to ratings of zero in SPSS(e.g.http://www.spssusers.co.uk/Events/2015/ALDRIDGE2015.pdf) The only solution I see is recoding the ratings to avoid the value of zero but, since I would have to keep in my dataset (which is very large!) the original ratings (which are the ones that are clinically meaningful), I would end up having so many variables that it would all get very confusing and it seems a pity, especially if the error message is due to me making some mistake I am unaware of. Any help would be greatly appreciated!!

I also have a second question: in clinical practice average values of the ordinal scales are often used. For instance if atrophy is 1 for the left hemisphere and 2 for the right hemisphere a value of 1.5 is frequently used to summarize this information. I have read the threads on means of ordinal variables and I understand this issue is controversial. However I need to calculate intra and inter-rater variability for the means of the ordinal scale. I can't use weighted kappa (which obviously works only with integers) and I would not want to use Spearman's correlation coefficient because I take it that it tells me how much the ratings are correlated in terms of a linear relationship but not how much they are identical to each other (i.e if one of the two raters systematically over-rated by one point, the scores of the two raters would have a correlation of one but would not be in agreement with each other which is what I am interested in). I think I could maybe use Cohen's unweighted kappa but I wonder if there is a better way to go about this..

I really would be very grateful to anyone who could give me a hand.

  • $\begingroup$ For Q2 why not multiply them all by 2? $\endgroup$ – mdewey Dec 22 '18 at 21:58
  • $\begingroup$ Thank you very much, it's a really good suggestion! I guess that would allow me to use weighted kappa without altering the properties of the data. Do you happen have any reference for this? (Sorry for being so fussy, it's just that some reviewers can be very difficult...). PS: Can someone help with the error message from SPSS: it's driving me mad! It is also a problem for calculating weighted kappa according to mdewey because in one of the two scales I have a couple of values which are zero (both ratings being zero).For the other scale multiplying by 2 works fine. $\endgroup$ – paola Dec 23 '18 at 9:25
  • $\begingroup$ For the error message you might be better asking on an SPSS forum as questions about using software are usually seen as off-topic here. You may also have to clarify what you mean by both rating being zero as that is unclear to me. Do you mean that is the code for them (which is not a problem)? Do you mean one rater did not use that code? Do you mean neither rater used that code (which is not a problem either)? $\endgroup$ – mdewey Dec 23 '18 at 11:55
  • $\begingroup$ The actual radiological scale has ratings that go from 0 (no atrophy) to 4 (severe atrophy) and is rated for both the right and left sides of the brain. Both ratings are zero if there in no atrophy in both the right and left sides so that the real mean for atrophy is zero (not the code, which, as you say, I can change). Thanks for following-up on the reply and also for the tip on the SPSS forum, I'll try straight away, it's just that is is very difficult to get answers (I have never managed to obtain any feedback from IBM's own forum on SPSS) $\endgroup$ – paola Dec 23 '18 at 12:52
  • $\begingroup$ I am afraid I still do not see the problem but if SPSS for some reason does not like the fact that atrophy can be zero then just add 1 to all the ratings. $\endgroup$ – mdewey Dec 24 '18 at 13:36

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.