Cohen's kappa is a measure of the degree to which 2 raters agree. There is also a test of inter-rater agreement based on kappa. Use [inter-rater] if you are interested in other aspects of IRA, but not this specific measure.

learn more… | top users | synonyms

15
votes
2answers
11k views

Inter-rater reliability for ordinal or interval data

Which inter-rater reliability methods are most appropriate for ordinal or interval data? I believe that "Joint probability of agreement" or "Kappa" are designed for nominal data. Whilst "Pearson" and ...
6
votes
1answer
2k views

Kappa statistic in plain English?

I'm reading a data mining book and it mentioned the Kappa statistic as a mean for evaluating the predication performance of classifiers. However I just can't understand this. I also checked wikipedia ...
2
votes
1answer
68 views

Calculating agreement between an ordinal and continuous scale

I'm having a bit of trouble wrapping my head around something. I have a data set which contains two columns that essentially attempt to measure the same thing, one on a 1-50 continuous scale and ...
2
votes
1answer
1k views

ICC and Kappa totally disagree

Much has been written on the ICC and Kappa, but there seems to be disagreement on the best measures to consider. My purpose is to identify some measure which shows whether there was agreement between ...
4
votes
2answers
2k views

Assessing and testing inter-rater agreement with kappa statistic on a set of binary and Likert items?

I am trying to calculate inter-rater reliability scores for 10 survey questions (on which there were 2 raters)--seven questions are binary (yes/no), and 3 are Likert-scale questions. Should ...
0
votes
0answers
190 views

Agreement statistics for confusion matrix with nominal categories

I have a decision support system that recommends Treatments (T1 to T8) to patients, where ...