Rater's agreement for each item

Question

I would like to calculate the agreement between 2 or more raters on all the judgments from a set, but also for each judgment.

For example:

rater1, rater2
aa, aa
bb, bb
cc, cd

I can use Cohen's Kappa or Fleiss Kappa to calculate the agreement for all the items, but can't find a way to calculate the agreement for each judgment. What R method should I use?

If I simply try to use Fleiss Kappa for the first row of the file ('aa', 'aa') I will get the following:

 Cohen's Kappa for 2 Raters (Weights: unweighted)

 Subjects = 1 
 Raters = 2 
 Kappa = NaN 

      z = NaN 
 p-value = NaN 

I would really need a function to do this agreement for each judgment.

Answer 1

You can calculate the percent observed agreement for a single item: $$ A=\sum_{k=1}^q \frac{r_k(r_k-1)}{r(r-1)} $$ Where $q$ is the total number of categories, $r$ is the total number of raters that assigned the item to any category, and $r_k$ is the number of raters that assigned the item to category $k$.

Answer 2

Instead of calculating rater's agreement for each judgment, I calculated rater's agreement for all the judgments that refer to the same subject, therefore I ended up with:

Fleiss Kappa for 2 raters
Subjects = 3
Raters = 2

There seems to be no Kappa or Fleiss Kappa that would give us the agreement for just 2 raters and 1 subject.