Cohen's kappa is a measure of the degree to which 2 raters agree. There is also a test of inter-rater agreement based on kappa. Use [inter-rater] if you are interested in other aspects of IRA, but not this specific measure.

learn more… | top users | synonyms

15
votes
2answers
10k views

Inter-rater reliability for ordinal or interval data

Which inter-rater reliability methods are most appropriate for ordinal or interval data? I believe that "Joint probability of agreement" or "Kappa" are designed for nominal data. Whilst "Pearson" and ...
3
votes
1answer
978 views

Kappa statistic in plain English?

I'm reading a data mining book and it mentioned the Kappa statistic as a mean for evaluating the predication performance of classifiers. However I just can't understand this. I also checked wikipedia ...
2
votes
1answer
58 views

Calculating agreement between an ordinal and continuous scale

I'm having a bit of trouble wrapping my head around something. I have a data set which contains two columns that essentially attempt to measure the same thing, one on a 1-50 continuous scale and ...
2
votes
1answer
911 views

ICC and Kappa totally disagree

Much has been written on the ICC and Kappa, but there seems to be disagreement on the best measures to consider. My purpose is to identify some measure which shows whether there was agreement between ...
4
votes
2answers
2k views

Assessing and testing inter-rater agreement with kappa statistic on a set of binary and Likert items?

I am trying to calculate inter-rater reliability scores for 10 survey questions (on which there were 2 raters)--seven questions are binary (yes/no), and 3 are Likert-scale questions. Should ...