1
$\begingroup$

I intend to ask a number of experts to match five out of a possible ten team roles (recorder, chairman and so on) to five distinct stages of a project.

I cannot seem to find any source for determining how many raters I need in order to achieve sufficient inter-rater reliability. Does anyone know of a reliable source for calculating the number of raters needed?

$\endgroup$
  • $\begingroup$ As far as I know, inter-rater reliability is unrelated to the number of raters you have. If you wanted to estimate the amount of inter-rater reliability that would exist in a population of raters, a larger sample should lead to narrower confidence intervals, but that's a different issue. If you just want your raters to be in closer agreement w/ each other, I might focus on polishing your materials &/or some training for your raters. $\endgroup$ – gung - Reinstate Monica Oct 23 '12 at 13:11
0
$\begingroup$

Usually there are only 2 raters in interrater reliability (although there can be more). You don't get higher reliability by adding more raters: Interrarter reliability is usually measure by either Cohen's $\kappa$ or a correlation coefficient.

You get higher reliability by having either better items or better raters.

The Wikipedia entry is not a bad place to start, and it has good references.

You may be confusing number of raters with number of items.

$\endgroup$
  • $\begingroup$ Thanks @Peter. I read the Wikipedia entry, along with some other sources, but I found it hard to believe the reliability of one's results doesn't increase with the number of judges agreeing: three people agreeing on something seems stronger than two people doing the same, don't you think? $\endgroup$ – daniel_ev Oct 23 '12 at 13:02
  • $\begingroup$ That is not the question addressed by interrater reliability, odd as it sounds. $\endgroup$ – Peter Flom - Reinstate Monica Oct 23 '12 at 20:36
0
$\begingroup$

I have read that when the number of raters increases from 2 to 4, the increase in reliability is quite significant.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.