0
$\begingroup$

I have about 10 raters giving weighted ratings to different items of a dataset. The number of raters may differ from item to another. An item may have any number of raters between 2 and 10. Raters can give each item one of 3 values (-1,0, and 1).

I tried using weighted kappa to measure the agreement between each pair of raters at a time. However, the results I am getting seem a little strange. When only 2 of the 3 values are used by the raters the result is 0 even if they agree on most items. Example:

(-1,-1; -1 ,-1;-1 , -1; -1, -1;-1 , 0; -1, 0)

even though the two raters gave the same rating to 4/6 items the result is 0.

Is cohen kappa the best suited method for my problem? If not, what's a better alternative.

Improve this question
$\endgroup$
1
$\begingroup$

In your example, Cohen's $\kappa$ coefficient is equal to $0$ despite observed agreement ($p_o$) being relatively high because chance agreement ($p_c$) is also high according to Cohen's assumptions.

$$ \kappa = \frac{p_o - p_c}{1 - p_c} = \frac{.667 - .667}{1 - .667} = .000 $$

You might try another chance-adjusted index that makes different assumptions than Cohen's $\kappa$ coefficient. One such option would be Bennett et al.'s $S$ score below, where $q$ is the number of possible categories. In this example, assuming the same three category options, $S$ would be higher.

$$ S = \frac{p_o - 1/q}{1 - 1/q} = \frac{.667 - .333}{1 - .333} = .500 $$

Both of these metrics (and several others) can be adapted to multiple raters, multiple categories, missing data, and weighting schemes for non-nominal categories. See my mReliability website.

Improve this answer
$\endgroup$
5
  • $\begingroup$ In this case, kappa and S are equal when the classed are balanced and differ to the extent to which the classes are imbalanced. $\endgroup$ – Jeffrey Girard Dec 30 '19 at 3:36
  • $\begingroup$ Balanced would mean that the frequency of each class is the same. So if you have three classes -1, 0, and 1, then the number of items in each class is the same. Let's say there are 12 items being rated, balanced would mean there are 4 items in each class. $\endgroup$ – Jeffrey Girard Dec 30 '19 at 3:43
  • $\begingroup$ Yes, this has been written about extensively in several different fields. An accessible treatment is doi.org/10.1016/0895-4356(90)90158-L $\endgroup$ – Jeffrey Girard Dec 30 '19 at 3:56
  • $\begingroup$ It would probably be best to start a new question to discuss these things. $\endgroup$ – Jeffrey Girard Dec 30 '19 at 4:09
  • $\begingroup$ Dear Jeffrey, I was wondering if you might possibly have had a chance to look at my reliability question? $\endgroup$ – rnorouzian Jan 1 '20 at 0:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.