I would like to calculate the agreement between 2 or more raters on all the judgments from a set, but also for each judgment.

For example:

rater1, rater2
aa, aa
bb, bb
cc, cd

I can use Cohen's Kappa or Fleiss Kappa to calculate the agreement for all the items, but can't find a way to calculate the agreement for each judgment. What R method should I use?

If I simply try to use Fleiss Kappa for the first row of the file ('aa', 'aa') I will get the following:

 Cohen's Kappa for 2 Raters (Weights: unweighted)

 Subjects = 1 
 Raters = 2 
 Kappa = NaN 

      z = NaN 
 p-value = NaN 

I would really need a function to do this agreement for each judgment.


2 Answers 2


You can calculate the percent observed agreement for a single item: $$ A=\sum_{k=1}^q \frac{r_k(r_k-1)}{r(r-1)} $$ Where $q$ is the total number of categories, $r$ is the total number of raters that assigned the item to any category, and $r_k$ is the number of raters that assigned the item to category $k$.

  • 2
    $\begingroup$ I marked the other Q. as a duplicate of this one (since this one has another answer). I'll remove your other answer rather than this one, even though it was posted earlier. Thanks for your help. $\endgroup$
    – Glen_b
    Apr 14, 2016 at 4:09

Instead of calculating rater's agreement for each judgment, I calculated rater's agreement for all the judgments that refer to the same subject, therefore I ended up with:

Fleiss Kappa for 2 raters
Subjects = 3
Raters = 2

There seems to be no Kappa or Fleiss Kappa that would give us the agreement for just 2 raters and 1 subject.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.