Inter-rater agreement: number of raters vs. number of test subjects

Question

I have a set of n texts. Each text is to be assigned a topic by i raters who choose the topic from a set of k discrete topic labels (A, B, C, ... K). So, for each text there are i ratings (categorical data):

On this dataset, I want to calculate inter-rater agreement. Given that there are more than two raters, Fleiss' kappa would be a reasonable choice, since Cohen's kappa measures agreement between two raters only.

Given that both the texts to be annotated as well as the raters who annotate the texts are costly, I need to limit the number of texts or the number of raters; otherwise the study would be too expensive.

Now my question: In view of this restriction, which one (number of raters or number of texts) contributes more to the robustness of the inter-rater reliability measure? In other words, is it better to limit the number n of texts to be annotated, or the number i of raters if my goal is to assess how generalisable the obtained results are. The overall goal of the study is to assess how useful the set of k discrete topic labels is for the task of text annotation. Note that the both the raters and the texts in the study are sampled from a potentially very large population of texts/raters. Besides that, the number k of categorical topic labels the raters can choose from is rather high (k equals approximately 200 to 300).

Answer 1

If you want to assess the ability of your results to generalize to different texts than those in your sample, then you will want to increase the number of texts. If you want to assess the ability of your results to generalize to different raters than those in your sample, then you will want to increase the number of raters. The relative balance of texts to raters will depend in part on the relative importance you place on the generalizability of each. In many domains, you will want to emphasize representation of the objects of measurement (e.g., texts) but it depends on the context and goals of the research.

One thing that may be helpful is to read up on Generalizability Theory. A Decision (D) Study would help you to quantify what would happen to the generalizability of results under different numbers of texts and raters.

• Brennan, R. L. (2001). Generalizability Theory. New York: Springer-Verlag.

• Cronbach, L.J., Gleser, G.C., Nanda, H., & Rajaratnam, N. (1972). The dependability of behavioral measurements: Theory of generalizability for scores and profiles. New York: John Wiley.

• Shavelson, R.J., & Webb, N.M. (1991). Generalizability Theory: A Primer. Thousand Oaks, CA: Sage.