Situation: I am unsure how many judges I need for good inter-rater reliability and how to calculate the inter-reliability in R depending on the number of judges.

Task: I am wanting to obtain people's judgements on whether a list of words contains event words (and if so how many).

Attempt: I have a list of words that have been randomly generated waiting to be judged. The instructions to the task sit at the top of the excel file, and judges will be scanning down words in column 1 and marking whether they judge it to be an event word in column 2. I have 5 potential candidates who have kindly volunteered their time.

Remaining Issue: I would like my decision as to the number of raters to be underscored by some psychometric evidence, but I'm a little lost regarding how to find and select this evidence. I have read (somewhere on Cross Validated) that less is better for inter-rater reliability, which makes sense. The 2 questions remain: (a) how many raters should I pick out of the 5 and (b) what inter-rater reliability statistic should I use?

  • 1
    $\begingroup$ Where did you read that that less is better for inter-rater reliability? Why does that make sense? $\endgroup$ Commented Aug 4, 2021 at 15:20
  • 1
    $\begingroup$ That's a good question @kjetilbhalvorsen. I read it from a non-reputable source on quora somewhere. I thought it made logical sense because less raters that also think similarly might provide more similar answers, therefore greater internal consistency reliability or correlation between items. Since then, I have thought more about it and it seems that more may be better for this task. Do you have any good sources or could you provide a perspective from a methods based pov? $\endgroup$
    – n.baes
    Commented Aug 6, 2021 at 13:48


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Browse other questions tagged or ask your own question.