# Why is Cohen's kappa low despite high observed agreement?

I have about 10 raters giving weighted ratings to different items of a dataset. The number of raters may differ from item to another. An item may have any number of raters between 2 and 10. Raters can give each item one of 3 values (-1,0, and 1).

I tried using weighted kappa to measure the agreement between each pair of raters at a time. However, the results I am getting seem a little strange. When only 2 of the 3 values are used by the raters the result is 0 even if they agree on most items. Example:

(-1,-1; -1 ,-1;-1 , -1; -1, -1;-1 , 0; -1, 0)

even though the two raters gave the same rating to 4/6 items the result is 0.

Is cohen kappa the best suited method for my problem? If not, what's a better alternative.

In your example, Cohen's $\kappa$ coefficient is equal to $0$ despite observed agreement ($p_o$) being relatively high because chance agreement ($p_c$) is also high according to Cohen's assumptions.

$$\kappa = \frac{p_o - p_c}{1 - p_c} = \frac{.667 - .667}{1 - .667} = .000$$

You might try another chance-adjusted index that makes different assumptions than Cohen's $\kappa$ coefficient. One such option would be Bennett et al.'s $S$ score below, where $q$ is the number of possible categories. In this example, assuming the same three category options, $S$ would be higher.

$$S = \frac{p_o - 1/q}{1 - 1/q} = \frac{.667 - .333}{1 - .333} = .500$$

Both of these metrics (and several others) can be adapted to multiple raters, multiple categories, missing data, and weighting schemes for non-nominal categories. See my mReliability website.

• In this case, kappa and S are equal when the classed are balanced and differ to the extent to which the classes are imbalanced. – Jeffrey Girard Dec 30 '19 at 3:36
• Balanced would mean that the frequency of each class is the same. So if you have three classes -1, 0, and 1, then the number of items in each class is the same. Let's say there are 12 items being rated, balanced would mean there are 4 items in each class. – Jeffrey Girard Dec 30 '19 at 3:43
• Yes, this has been written about extensively in several different fields. An accessible treatment is doi.org/10.1016/0895-4356(90)90158-L – Jeffrey Girard Dec 30 '19 at 3:56
• It would probably be best to start a new question to discuss these things. – Jeffrey Girard Dec 30 '19 at 4:09
• Dear Jeffrey, I was wondering if you might possibly have had a chance to look at my reliability question? – rnorouzian Jan 1 '20 at 0:13