I am trying to calculate inter-rater reliability scores for 10 survey questions (on which there were 2 raters)--seven questions are binary (yes/no), and 3 are Likert-scale questions.
Should inter-rater reliability be tested on EACH of the 10 questions, or is there an overall inter-relater reliability test that tests reliability of all questions at one time? If so, what is it?
For the binary questions, the agreement level between the two raters is 70-90% on nearly every question, however the Kappa score is often very poor (0.2- 0.4). Can this be right? (And if so, is there a more appropriate test?)
Finally, can you use a Kappa-based test on Likert scale questions? If not, what is the correct test for inter-rater reliability?