Kappa (interrater agreement) alternative for count-based data which results in inflated agreement at lower counts?
I have a data set where four coders are rating 800 items on various attributes. This is achieved by reporting a count of the prevalence of each attribute, e.g. rater 1 thinks attribute A appears in ...
I used the following R script to calculate the Kappa and ICC for a measured done by two raters on 23 subjects: ...
The Kappa ($\kappa$) statistic was introduced in 1960 by Cohen  to measure agreement between two raters. Its variance, however, had been a source of contradictions for quite a some time. My ...
I have a data set of 11,000+ distinct items, each of which was classified on a nominal scale by at least 3 different raters on Amazon's Mechanical Turk. 88 different raters provided judgments for ...
Which inter-rater reliability methods are most appropriate for ordinal or interval data? I believe that "Joint probability of agreement" or "Kappa" are designed for nominal data. Whilst "Pearson" and ...