0
$\begingroup$

I've a dataset with items for which some users have given a score from 1 to 6. I wanted to compute the Fleiss' kappa to get the inter user agreement. The thing is that a I want a weighted version of this measure, because the agreement between a 5 and a 6 is a bigger than a 2 and a 6.

There exists a measure that can do this?

Thanks a lot

$\endgroup$

1 Answer 1

3
$\begingroup$

Gwet (2014) introduced a generalized version of Scott's pi coefficient, which is equivalent to Fleiss' kappa when applied with nominal weights. I recommend you read more about it in Gwet (2014). I'm also providing the formula here:

$$\pi=\frac{p_o-p_c}{1-p_c}$$

$$p_o=\frac{1}{n'}\sum_{i=1}^n\sum_{k=1}^q\frac{r_{ik}(r_{ik}^\star-1)}{r_i(r_i-1)}$$

$$r_{ik}^\star=\sum_{l=1}^qw_{kl}r_{il}$$

$$p_c= \sum_{k,l}^qw_{kl}\pi_k\pi_l$$

$$\pi_k=\frac{1}{n}\sum_{i=1}^n\frac{r_{ik}}{r_i}$$

$\pi$ is the chance-adjusted reliability index

$n'$ is the number of items that were assigned to any category by two or more raters

$n$ is the total number of items

$q$ is the total number of categories

$r_{ik}$ is the number of raters that assigned item $i$ to category $k$

$r_i$ is the number of raters that assigned item $i$ to any category

$w_{kl}$ is the weight associated with two raters assigning an item to categories $k$ and $l$ respectively

There are different weighting schemes available. More information on weighting schemes is available here, and more information on the generalized pi coefficient is available here.

References:

Gwet, K. L. (2014). Handbook of inter-rater reliability: The definitive guide to measuring the extent of agreement among raters (4th ed.). Gaithersburg, MD: Advanced Analytics.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge that you have read and understand our privacy policy and code of conduct.

Not the answer you're looking for? Browse other questions tagged or ask your own question.