How can I measure inter-rater agreement, e.g., using the Kappa coefficient, during an exploratory study where the categories are not predetermined?

  • $\begingroup$ Could you provide more information on your setup? Are raters going to freely assess some subjects or topics, like if "categories" were open-ended questions? $\endgroup$
    – chl
    Nov 19 '20 at 18:01
  • $\begingroup$ @chi That's correct. It is ultimately open-ended. $\endgroup$ Nov 19 '20 at 18:43
  • $\begingroup$ Any way to recode the open-ended answers? I f so you could end up with a fixed number of categories, which could be handled with inter-agreement methods after you discard the less frequent ones. $\endgroup$
    – chl
    Nov 19 '20 at 20:49
  • $\begingroup$ I guess we could do a pilot study (e.g., 5%) to come up with a fixed number of categories. $\endgroup$ Nov 19 '20 at 21:21
  • 1
    $\begingroup$ Sure, that would be great. Otherwise I don't see how you could possibly use Cohen's kappa or any other measure of agreement if the number of items is not fixed. $\endgroup$
    – chl
    Nov 20 '20 at 15:04

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.