I am trying to calculate interrater reliability in SPSS for both pre and post test of the same measure that is administered as part of a prison intake program. The measure has 20 items and then a total score. I am just running on total score agreement.

I ran crosstabs and calculated kappa for the pretest and correlations. Is that all I need to do? Is there more that would be helpful to report for publication?

Also, I have one rater that I suspect is a problem. I also want to use this information for training raters. How can I identify how error associates with an individual rater?

If you could tell me your thoughts for SPSS, that would be helpful. I am learning R, but am currently completely R illiterate.

Thanks for any information.


If you are looking at inter-rater reliability on the total scale scores (and you should be), then Kappa would not be appropriate. If you have two raters for the pre-test and two for the post-test, then a correlation would be informative. If you have more than two raters, computing the ICC (intraclass correlation) from the SPSS RELIABILITY procedure would be appropriate.

  • 1
    $\begingroup$ Not sure I understand the part about correlation on pre/post measurements: linear or rank-based correlation provides no information on the level of agreement between two raters. $\endgroup$ – chl Jan 20 '13 at 9:20
  • $\begingroup$ Is intra class fine for inter rater reliability? $\endgroup$ – Cesare Camestre Jul 18 '13 at 21:55

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.