1
$\begingroup$

I am trying to calculate the Fleiss' Kappa of the following data and I got the result:

 [[0 0 3]
 [0 1 2]
 [0 0 3]
 [0 0 3]
 [0 0 3]
 [0 0 3]
 [0 0 3]
 [0 0 3]
 [0 0 3]
 [0 1 2]
 [0 0 3]
 [0 0 3]
 [0 0 3]
 [0 0 3]
 [0 0 3]
 [0 0 3]
 [0 0 3]
 [0 1 2]
 [0 0 3]
 [0 0 3]
 [0 0 3]
 [0 0 3]
 [0 0 3]
 [0 0 3]
 [0 0 3]
 [0 0 3]
 [0 0 3]]
3 raters.
27 subjects.
3 categories.
p = [0.0, 0.037037037037037035, 0.9629629629629629]
P = [1.0, 0.3333333333333333, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.3333333333333333, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 0.3333333333333333, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0, 1.0]
Pbar = 0.925925925926
PbarE = 0.928669410151
kappa = -0.0384615384615

However, the ratings of the 3 raters obviously have huge agreement: nearly all of the 27 items got rating "3".

I am really confused...Can someone explain this to me please?

|cite|improve this question
$\endgroup$
2
$\begingroup$

I believe that calculations for Fleiss' Kappa would view the way your data set is set out as having disagreement between raters. I.e. the first row says rater one recorded 0, rater two recorded 0, and rater three recorded 3 = only two out of three of the raters agreed. The second row shows that rater one (column one answers), rater two (column two answers) and rater three (column three answers) all recorded a different result, therefore there is no agreement between them. You may find this helpful for setting up your data set http://dfreelon.org/utils/recalfront/recal3/ or this http://www.real-statistics.com/reliability/fleiss-kappa/

|cite|improve this answer
$\endgroup$
  • $\begingroup$ I think I understand what you say. Thanks! I'll think about it! $\endgroup$ – Zhiya Sep 22 '15 at 2:01
  • $\begingroup$ I have to disagree with this answer. The values within the matrices are the number of raters who agreed in that category. Assuming column 0 represents category 0, then the first 1x3 matrix [0 0 3] would translate to: All 3 raters agreed that it is the third category. The numbers (0 or 3) are not treated as a nominal or categorical value; They are the number who agreed under that category represented under that column. Your last reference even noted it. See: let xij = the number of judges that assign category j to subject i. $\endgroup$ – iwillnot Feb 28 '18 at 12:57

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.