I would like to calculate the agreement between 2 or more raters on all the judgments from a set, but also for each judgment.
rater1, rater2 aa, aa bb, bb cc, cd
I can use Cohen's Kappa or Fleiss Kappa to calculate the agreement for all the items, but can't find a way to calculate the agreement for each judgment. What R method should I use?
If I simply try to use Fleiss Kappa for the first row of the file ('aa', 'aa') I will get the following:
Cohen's Kappa for 2 Raters (Weights: unweighted) Subjects = 1 Raters = 2 Kappa = NaN z = NaN p-value = NaN
I would really need a function to do this agreement for each judgment.