1
$\begingroup$

In Measuring nominal scale agreement among many raters, where the authors define Fleiss kappa, the agreement by chance is defined as

enter image description here

where $p_j$ is defined as the proportion of ratings which were to the $j$th category.

Can someone explain how the authors arrived at this equation?

An additional reference for the equations can be found on Wikipedia

$\endgroup$

1 Answer 1

0
$\begingroup$

Say two raters respond randomly, with probability of yes = 0.9 and probability of no = 0.1, we can put the results into this table.

      Yes    |   No  
Yes          |        |  0.9 
 No          |        |  0.1
---------------------------
Total   0.9  |   0.1  |  1.0

Best case (perfect agreement):

            R1 
R2     Yes   |   No  
Yes    0.9   |   0.0  |  0.9 
 No    0.0   |   0.1  |  0.1
---------------------------
Total   0.9  |   0.1  |  1.0

Worst case (max disagreement):

            R1 
 R2    Yes   |   No  
 Yes   0.81  |   0.09  |  0.9 
 No    0.09  |   0.01  |  0.1
---------------------------
Total   0.9  |   0.1   |  1.0

How much disagreement do we have in the worst case? 0.9^2 + 0.1^2 = 0.82

That's the formula.

$\endgroup$
2
  • $\begingroup$ The way I am thinking about it is to calculate the probability that any two raters randomly agree. Assuming the categories are mutually independent $P(\textrm{random chance agreement of any pair}) = \sum_j P(\textrm{random chance agreement of any pair}) = \sum_j p_j \cdot p_j$ where $p_j$ is the probability of assigning the unit to category $j$. $\endgroup$
    – crabnebul
    Commented Mar 2, 2023 at 21:26
  • $\begingroup$ Yep, that makes sense. $\endgroup$ Commented Mar 3, 2023 at 3:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.