1
$\begingroup$

In Measuring nominal scale agreement among many raters, where the authors define Fleiss kappa, the agreement by chance is defined as

enter image description here

where $p_j$ is defined as the proportion of ratings which were to the $j$th category.

Can someone explain how the authors arrived at this equation?

An additional reference for the equations can be found on Wikipedia

Improve this question
$\endgroup$

1 Answer 1

Reset to default
0
$\begingroup$

Say two raters respond randomly, with probability of yes = 0.9 and probability of no = 0.1, we can put the results into this table.

      Yes    |   No  
Yes          |        |  0.9 
 No          |        |  0.1
---------------------------
Total   0.9  |   0.1  |  1.0

Best case (perfect agreement):

            R1 
R2     Yes   |   No  
Yes    0.9   |   0.0  |  0.9 
 No    0.0   |   0.1  |  0.1
---------------------------
Total   0.9  |   0.1  |  1.0

Worst case (max disagreement):

            R1 
 R2    Yes   |   No  
 Yes   0.81  |   0.09  |  0.9 
 No    0.09  |   0.01  |  0.1
---------------------------
Total   0.9  |   0.1   |  1.0

How much disagreement do we have in the worst case? 0.9^2 + 0.1^2 = 0.82

That's the formula.

Improve this answer
$\endgroup$
2
  • $\begingroup$ The way I am thinking about it is to calculate the probability that any two raters randomly agree. Assuming the categories are mutually independent $P(\textrm{random chance agreement of any pair}) = \sum_j P(\textrm{random chance agreement of any pair}) = \sum_j p_j \cdot p_j$ where $p_j$ is the probability of assigning the unit to category $j$. $\endgroup$
    – crabnebul
    Mar 2 at 21:26
  • $\begingroup$ Yep, that makes sense. $\endgroup$
    – Jeremy Miles
    Mar 3 at 3:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.