# Chance Agreement for Fleiss Kappa

In Measuring nominal scale agreement among many raters, where the authors define Fleiss kappa, the agreement by chance is defined as where $$p_j$$ is defined as the proportion of ratings which were to the $$j$$th category.

Can someone explain how the authors arrived at this equation?

An additional reference for the equations can be found on Wikipedia

Say two raters respond randomly, with probability of yes = 0.9 and probability of no = 0.1, we can put the results into this table.

      Yes    |   No
Yes          |        |  0.9
No          |        |  0.1
---------------------------
Total   0.9  |   0.1  |  1.0


Best case (perfect agreement):

            R1
R2     Yes   |   No
Yes    0.9   |   0.0  |  0.9
No    0.0   |   0.1  |  0.1
---------------------------
Total   0.9  |   0.1  |  1.0


Worst case (max disagreement):

            R1
R2    Yes   |   No
Yes   0.81  |   0.09  |  0.9
No    0.09  |   0.01  |  0.1
---------------------------
Total   0.9  |   0.1   |  1.0


How much disagreement do we have in the worst case? 0.9^2 + 0.1^2 = 0.82

That's the formula.

• The way I am thinking about it is to calculate the probability that any two raters randomly agree. Assuming the categories are mutually independent $P(\textrm{random chance agreement of any pair}) = \sum_j P(\textrm{random chance agreement of any pair}) = \sum_j p_j \cdot p_j$ where $p_j$ is the probability of assigning the unit to category $j$. Mar 2 at 21:26
• Yep, that makes sense. Mar 3 at 3:08