7
$\begingroup$

Can somebody explain in-detailed differences between Fleiss kappa and Cohen kappa? And how the metric works under the hood?

  • When would one use Fleiss kappa over Cohen kappa?
  • What are the advantages/disadvantages of using Fleiss kappa over Cohen kappa?
Improve this question
$\endgroup$

1 Answer 1

Reset to default
6
$\begingroup$

Fleiss' $\kappa$ works for any number of raters, Cohen's $\kappa$ only works for two raters; in addition, Fleiss' $\kappa$ allows for each rater to be rating different items, while Cohen's $\kappa$ assumes that both raters are rating identical items.

However, Fleiss' $\kappa$ can lead to paradoxical results (see e.g. Gwet, Handbook of Interrater Reliability, namely that, even with nominal categories, reordering the categories can change the results. But Cohen's version has its own problems and can lead to odd results when there are large differences in the prevalence of possible outcomes (see e.g. Feinstein and Cicchetti, High Agreement but low Kappa.

Gwet's AC1 statistic appears to be immune to these problems. For R raters it is given by

$\gamma_1 = \frac{P_a-P_{e|\gamma_1}}{1-P_{e|\gamma_1}} $

where $P_{e|\gamma_1} = \frac{1}{K-1}\sum{\hat{\pi}_k}(1-\hat{\pi}_k)$

and $\hat{\pi}_k = \sum{\frac{R_{ik}}{R}} $

Improve this answer
$\endgroup$
1
  • 1
    $\begingroup$ Gwet, Handbook of Interrater Reliability - this link seems broken $\endgroup$
    – Łukasz Deryło
    Jun 1, 2020 at 5:59

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.