9
$\begingroup$

Can somebody explain in-detailed differences between Fleiss kappa and Cohen kappa? And how the metric works under the hood?

  • When would one use Fleiss kappa over Cohen kappa?
  • What are the advantages/disadvantages of using Fleiss kappa over Cohen kappa?
$\endgroup$

2 Answers 2

8
$\begingroup$

Fleiss' $\kappa$ works for any number of raters, Cohen's $\kappa$ only works for two raters; in addition, Fleiss' $\kappa$ allows for each rater to be rating different items, while Cohen's $\kappa$ assumes that both raters are rating identical items.

However, Fleiss' $\kappa$ can lead to paradoxical results (see e.g. Gwet, Handbook of Interrater Reliability, namely that, even with nominal categories, reordering the categories can change the results. But Cohen's version has its own problems and can lead to odd results when there are large differences in the prevalence of possible outcomes (see e.g. Feinstein and Cicchetti, High Agreement but low Kappa.

Gwet's AC1 statistic appears to be immune to these problems. For R raters it is given by

$\gamma_1 = \frac{P_a-P_{e|\gamma_1}}{1-P_{e|\gamma_1}} $

where $P_{e|\gamma_1} = \frac{1}{K-1}\sum{\hat{\pi}_k}(1-\hat{\pi}_k)$

and $\hat{\pi}_k = \sum{\frac{R_{ik}}{R}} $

$\endgroup$
1
  • 1
    $\begingroup$ Gwet, Handbook of Interrater Reliability - this link seems broken $\endgroup$ Commented Jun 1, 2020 at 5:59
0
$\begingroup$

Sadly I can not comment due to low reputation. However, I found the link to the aforementioned Handbook of Interrater Reliability (and other resources) and wanted to share: https://agreestat.com/books/default.html. I was able to find it by googling for the Handbook of Interrater Reliability in combination with "+Gwet" to show results linked to the author. It seems like the ebook is still updated and sold, the current volume is the 5th, ISBN is 9781792354649.

$\endgroup$
2
  • 2
    $\begingroup$ While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes. - From Review $\endgroup$ Commented Dec 6, 2023 at 8:26
  • $\begingroup$ Thank you for the feedback. I tried changing my initital answer to include your feedback. $\endgroup$
    – BrianBrain
    Commented Dec 6, 2023 at 14:14

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.