I have 260 teacher conferences to score using a rubric. How do two raters need to score these for rater agreement? Do we need three raters?
The use of a rubric is there to try to control for inter-rater variability. You can further decrease inter-rater variability by 'training' the raters (e.g. show all the raters the same video and tell them what they should score the video). Of course, in terms of getting a more precise estimate of the true rating, the more raters the better
With 2 raters only you have no way of knowing which rater is more accurate. With 3 raters or more you can use factor analysis to account for raters precision.