I have data on two processes, where the process assigns elements into ordered bins. I am interested in testing for agreement between the processes. What is the best way to do this (R code)? Here is ...
I have a data set of 11,000+ distinct items, each of which was classified on a nominal scale by at least 3 different raters on Amazon's Mechanical Turk. 88 different raters provided judgments for ...
Assessing and testing inter-rater agreement with kappa statistic on a set of binary and Likert items?
I am trying to calculate inter-rater reliability scores for 10 survey questions (on which there were 2 raters)--seven questions are binary (yes/no), and 3 are Likert-scale questions. Should ...
Which inter-rater reliability methods are most appropriate for ordinal or interval data? I believe that "Joint probability of agreement" or "Kappa" are designed for nominal data. Whilst "Pearson" and ...