Fleiss' Kappa alternative for ranking

Question

I've a dataset with items for which some users have given a score from 1 to 6. I wanted to compute the Fleiss' kappa to get the inter user agreement. The thing is that a I want a weighted version of this measure, because the agreement between a 5 and a 6 is a bigger than a 2 and a 6.

There exists a measure that can do this?

Thanks a lot

Answer 1

Gwet (2014) introduced a generalized version of Scott's pi coefficient, which is equivalent to Fleiss' kappa when applied with nominal weights. I recommend you read more about it in Gwet (2014). I'm also providing the formula here:

$$\pi=\frac{p_o-p_c}{1-p_c}$$

$$p_o=\frac{1}{n'}\sum_{i=1}^n\sum_{k=1}^q\frac{r_{ik}(r_{ik}^\star-1)}{r_i(r_i-1)}$$

$$r_{ik}^\star=\sum_{l=1}^qw_{kl}r_{il}$$

$$p_c= \sum_{k,l}^qw_{kl}\pi_k\pi_l$$

$$\pi_k=\frac{1}{n}\sum_{i=1}^n\frac{r_{ik}}{r_i}$$

$\pi$ is the chance-adjusted reliability index

$n'$ is the number of items that were assigned to any category by two or more raters

$n$ is the total number of items

$q$ is the total number of categories

$r_{ik}$ is the number of raters that assigned item $i$ to category $k$

$r_i$ is the number of raters that assigned item $i$ to any category

$w_{kl}$ is the weight associated with two raters assigning an item to categories $k$ and $l$ respectively

There are different weighting schemes available. More information on weighting schemes is available here, and more information on the generalized pi coefficient is available here.

References:

Gwet, K. L. (2014). Handbook of inter-rater reliability: The definitive guide to measuring the extent of agreement among raters (4th ed.). Gaithersburg, MD: Advanced Analytics.