Determining the number of raters for inter-rater reliability I intend to ask a number of experts to match five out of a possible ten team roles (recorder, chairman and so on) to five distinct stages of a project. 
I cannot seem to find any source for determining how many raters I need in order to achieve sufficient inter-rater reliability. Does anyone know of a reliable source for calculating the number of raters needed?
 A: Usually there are only 2 raters in interrater reliability (although there can be more). You don't get higher reliability by adding more raters: Interrarter reliability is usually measure by either Cohen's $\kappa$ or a correlation coefficient. 
You get higher reliability by having either better items or better raters. 
The Wikipedia entry is not a bad place to start, and it has good references.
You may be confusing number of raters with number of items. 
A: Inter-rater reliability does not increase with more raters because inter-rater reliability is (loosely) the average agreement between raters. Additional raters do not change how often (or the degree to which) raters agree. Inter-rater reliability is instead affected by the skill of the raters (relative to the difficulty of the task) and the degree to which raters are making the same assessment (i.e., if raters understand the task in the same way).
However, more raters will increase the reliability of the average rating, which is presumably what the OP was wondering. That is, if you have a single rater, the correlation between their ratings and the average rating of the population of potential raters would be the inter-rater reliability of the task. However, if we had more than one rater, the correlation between the average of their ratings and the population of potential raters would be greater than the inter-rater reliability. Thus, if inter-rater reliability is lower than required, ratings from multiple raters can be aggregated to reduce the noise of individual ratings.
A better question might then be, "How many raters are required to get an aggregate rater reliability of x, if inter-rater reliability is y?"
