1
$\begingroup$

I am trying to calculate the Fleiss' Kappa of the following data and I got the result:

 [[0 0 3]
 [0 1 2]
 [0 0 3]
 [0 0 3]
 [0 0 3]
 [0 0 3]
 [0 0 3]
 [0 0 3]
 [0 0 3]
 [0 1 2]
 [0 0 3]
 [0 0 3]
 [0 0 3]
 [0 0 3]
 [0 0 3]
 [0 0 3]
 [0 0 3]
 [0 1 2]
 [0 0 3]
 [0 0 3]
 [0 0 3]
 [0 0 3]
 [0 0 3]
 [0 0 3]
 [0 0 3]
 [0 0 3]
 [0 0 3]]
3 raters.
27 subjects.
3 categories.
p = [0.0, 0.037037037037037035, 0.9629629629629629]
P = [1.0, 0.3333333333333333, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.3333333333333333, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.3333333333333333, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
Pbar = 0.925925925926
PbarE = 0.928669410151
kappa = -0.0384615384615

However, the ratings of the 3 raters obviously have huge agreement: nearly all of the 27 items got rating "3".

I am really confused...Can someone explain this to me please?

$\endgroup$

2 Answers 2

2
$\begingroup$

I believe that calculations for Fleiss' Kappa would view the way your data set is set out as having disagreement between raters. I.e. the first row says rater one recorded 0, rater two recorded 0, and rater three recorded 3 = only two out of three of the raters agreed. The second row shows that rater one (column one answers), rater two (column two answers) and rater three (column three answers) all recorded a different result, therefore there is no agreement between them. You may find this helpful for setting up your data set http://dfreelon.org/utils/recalfront/recal3/ or this http://www.real-statistics.com/reliability/fleiss-kappa/

$\endgroup$
2
  • $\begingroup$ I think I understand what you say. Thanks! I'll think about it! $\endgroup$
    – Zhiya
    Sep 22, 2015 at 2:01
  • 1
    $\begingroup$ I have to disagree with this answer. The values within the matrices are the number of raters who agreed in that category. Assuming column 0 represents category 0, then the first 1x3 matrix [0 0 3] would translate to: All 3 raters agreed that it is the third category. The numbers (0 or 3) are not treated as a nominal or categorical value; They are the number who agreed under that category represented under that column. Your last reference even noted it. See: let xij = the number of judges that assign category j to subject i. $\endgroup$
    – dsapalo
    Feb 28, 2018 at 12:57
0
$\begingroup$

That also confused me at the beginning, but now I understand. When raters have the same answer, the probability of any given rater getting the same result as any of the other raters is high, that would be the "expected agreement". So, let's say, if two hipothetical raters gave the same answer for every case, then the probability for rater 1 giving the same answer as rater 2 would be 100% x 100% = 100%, so in this hipothetical scenario the expected agreement is 100%, they agree but probably only because of the high probability of answering the same thing, not because they have the same opinion. And Kappa does tell the level of agreement, but substracting that level of expected agreement. =)

So the more answers that are concentrated in one column, the more the expected probability, and you dont want that. The more answers soncentrated in the same column for any given row, the more the observed agreement, and that's what you want. Finally, the more difference between the expected probability and the observed probability, the better.

I hope I explained myself. Good luck!

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.