I used the following R script to calculate the Kappa and ICC for a measured done by two raters on 23 subjects: ...
The Kappa ($\kappa$) statistic was introduced in 1960 by Cohen  to measure agreement between two raters. Its variance, however, had been a source of contradictions for quite a some time. My ...
I have a data set of 11,000+ distinct items, each of which was classified on a nominal scale by at least 3 different raters on Amazon's Mechanical Turk. 88 different raters provided judgments for ...
Which inter-rater reliability methods are most appropriate for ordinal or interval data? I believe that "Joint probability of agreement" or "Kappa" are designed for nominal data. Whilst "Pearson" and ...