I have 260 teacher conferences to score using a rubric. How do two raters need to score these for rater agreement? Do we need three raters?
The use of a rubric is there to try to control for inter-rater variability. You can further decrease inter-rater variability by 'training' the raters (e.g. show all the raters the same video and tell them what they should score the video). Of course, in terms of getting a more precise estimate of the true rating, the more raters the better