# Determining the number of raters for inter-rater reliability

I intend to ask a number of experts to match five out of a possible ten team roles (recorder, chairman and so on) to five distinct stages of a project.

I cannot seem to find any source for determining how many raters I need in order to achieve sufficient inter-rater reliability. Does anyone know of a reliable source for calculating the number of raters needed?

• As far as I know, inter-rater reliability is unrelated to the number of raters you have. If you wanted to estimate the amount of inter-rater reliability that would exist in a population of raters, a larger sample should lead to narrower confidence intervals, but that's a different issue. If you just want your raters to be in closer agreement w/ each other, I might focus on polishing your materials &/or some training for your raters. – gung - Reinstate Monica Oct 23 '12 at 13:11

Usually there are only 2 raters in interrater reliability (although there can be more). You don't get higher reliability by adding more raters: Interrarter reliability is usually measure by either Cohen's $\kappa$ or a correlation coefficient.