I have an experiment where 4 raters gave their responses to 4 stimuli, and I need to calculate the Fleiss Kappa to check the agreements of the raters. However, I get strange results from the R function implementing the Fleiss analysis.

Participant1 <- c(16, 15, 16, 16)
Participant2 <- c(16, 16, 16, 16)
Participant3 <- c(16, 16, 16, 16)
Participant4 <- c(16, 16, 16, 15)
data <- data.frame(Participant1, Participant2, Participant3, Participant4)

The output is

> data
  Participant1 Participant2 Participant3 Participant4
1           16           16           16           16
2           15           16           16           16
3           16           16           16           16
4           16           16           16           15

> kappam.fleiss(data)
 Fleiss' Kappa for m Raters

 Subjects = 4 
  Raters = 4 
   Kappa = -0.143 

        z = -0.7 
  p-value = 0.484 

The value for kappa is negative and with a non-significant p-value, despite a clear agreement between raters. Why? Personally, I do not really get the answer to the similar question reported here: Strange values of Cohen's kappa

So, why is the Fleiss analysis useful? The results seem to me to not give an indication on how much raters agreed.

How can I simply calculate the agreement between the four raters?


1 Answer 1


The problem is that there is almost no variation among the raters and the tiny bit of variation that does exist is not in agreement. There are only two ratings that are not 16 and they are for different cases, so you get a negative kappa. That's a correct result. You may, however, want a different measure.

  • $\begingroup$ Many thanks Peter. So, which is the correct analysis I have to perform? Of course I just put that table as an example, but I have other tables to analyze, where it is clear that there is an agreement between rathers, but I need to quantify it. Any suggestion? $\endgroup$
    – L_T
    Oct 31, 2017 at 15:33
  • $\begingroup$ For the table you gave, you could just say that nearly all the ratings were 16 and the two that were not were 15. You could put it in a frequency table (a one way table) or you could give the mean and sd. $\endgroup$
    – Peter Flom
    Nov 1, 2017 at 12:43

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.