Inconsistency of findings between Kappa and ICC for IRR study

Question

I used the following R script to calculate the Kappa and ICC for a measured done by two raters on 23 subjects:

library(irr)
temp <- structure(list(value.x = c(10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 2, 10, 10, 2), 
                       value.y = c(8, 8, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 8, 10, 10, 10, 10, 5, 10, 8, 5)),
                  .Names = c("value.x", "value.y"), row.names = c(NA, -23L), class = "data.frame")

kappa2(temp[,c("value.x","value.y")],weight="squared")

 Cohen's Kappa for 2 Raters (Weights: squared)

 Subjects = 23 
   Raters = 2 
    Kappa = 0.121 

        z = 1.61 
  p-value = 0.106 

icc(temp[,c("value.x","value.y")],model="twoway",type="agreement",unit="average")

 Average Score Intraclass Correlation

   Model: twoway 
   Type : agreement 

   Subjects = 23 
     Raters = 2 
   ICC(A,2) = 0.892

 F-Test, H0: r0 = 0 ; H1: r0 > 0 
 F(22,22.2) = 8.98 , p = 1.27e-06 

 95%-Confidence Interval for ICC Population Values:
  0.746 < ICC < 0.954

The measurement is actually a 5-point scale which variables are c(0,2,5,8,10).

I am wondering why Kappa and ICC gives so much different result: Kappa concluded that there is poor agreement, but ICC concluded that there is excellent agreement.

Which one should I believe?

Answer 1

I think a plot of your data is revealing:

value.x = c(10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10,
            10, 10, 10, 10, 10, 10, 2, 10, 10, 2)
value.y = c(8, 8, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10, 10,
            8, 10, 10, 10, 10, 5, 10, 8, 5)
plot(jitter(value.x), jitter(value.y))

The ICC and kappa are going to be differently influenced by the bimodal nature of both ratings, particularly for rater x, who rated all things 2 or 10.

If "low" vs. "high" is all you are concerned about, there is nearly perfect agreement. But if the fact that x gave some people 10 who y gave 8, and x gave some people 2 who y gave 5 is of interest, then the agreement is not so good.

This could be seen even more clearly in a plot of just the values over 5:

ratings <- data.frame(cbind(value.x, value.y))
subset <- subset(ratings, value.x > 2 & value.y > 2)

with(subset, plot(jitter(value.x), jitter(value.y)))

among the high ratings, it's pretty "blobby".

Answer 2

Rater 1 is only giving two values - very low (2) and extremely high (10). Rater 2 gives three values mid (5), high (8) and extremely high (10).

In view of the different ranges, you might consider a rank/Spearman correlation if you believe this is a matter of mismatch of scales, or direct/Pearson correlation if you believe they have the same idea of scales and both do use the full scale when appropriate. You could also map them to a Low, Medium, High scale where 8 or Medium is interpreted as sitting on the fence or abstaining by Rater 2, and Rater 1 never abstains. If we ignore the fence-sitting cases, it is now dichotomous. Note that there is perfect agreement on the Lows, just not on their levels.

If we are rating students, Rater 2 would be saying all students deserve to pass, and there are a stack of HD students, a few DN students and a couple of PASS students. Rater 1 is highly bimodal saying two deserve to FAIL outright, and the rest deserve full marks, HDs. The ranking should not be used in such a case as they are meant to be marked on the same scale, but different standards of marking would need to be reconciled before trying to combine the marks.

Understanding the dichotomous form of the problem (abstentions ignored as missing data) gives insights into your question. Let's suppose they agree yes on A, disagree one way on B, the other on C, and agree no on D of the examples:
A | B
- - -
C | D

With Mid=8 we get A=17, B=C=0, D = 2, and all Kappa and Correlation estimates will agree on 1. But, with Mid=5, 8 as High/Mid (discarding two with 5) we get B=C=D=0, Kappa=Correlation=0.
So the question boils down to how much we regard specific values as equivalent or Mid!

The determinant dtp = AD-BC sets the sign of the correlation (assuming all raters to give a spread and thus implicitly define a scale), and the Prev and IPrev represents the marginal prevalences of Rater 1 (what proportion of Highs resp Lows - the Gold Standard or Real World in my work cited below) and Bias represents the same for Rater 2 (the Predictions in my work). Below KC and KF refer to Cohen and Fleiss Kappa, while C is (Matthews) Correlation and B and M are the two (equations 5 to 9 from the ref. below)

KC = dtp/[(Prev*IBias+Bias*IPrev)/2].
KF = dtp/[(Prev+Bias)*(IBias+IPrev)/4].
B = dtp/[(Prev*IPrev)].
M = dtp/[(Bias*IBias)].
C = dtp/[√(Prev*IPrev*Bias*IBias)].

The reference relates to the use of these for Machine Learning rather than Human Raters, and the multivalued case (greater than two values) is more complex but generalizes this result. Treating as a 10 or 11 class problem (1..10 or 0..10) will also give 0s as there is only agreement for 10, so there is no meaningful basis for comparison.

So we can see the effect of different biases and that scaling choices place different expectations on the similarity of the biases of the two raters (Bias and Prev, and their inverses). Note in particular what happens as any of the four biases approaches 0, with the denominator approaching zero for the regression/correlation normalizations, but with two near zero biases needed to achieve this with the kappa formulations (note that if any bias is zero, viz. there is no spread for that rater, dtp is zero - the limits remain in the [0,1] range).

In summary it is largely a matter of consistency of scale and how we treat (apparent) differences in bias. I remain to be convinced that Kappa is a reasonable statistic, and it seems there is no valid probabilistic interpretation for either Cohen or Fleiss Kappa, but I am convinced for B, C and M, and would recommended Spearman Correlation for rater agreement, Pearson Correlation for even-hand cases where there is reason to believe they are on the same scale, Matthews Correlation for working with dichotomous contingency, given we see no reason to believe that one is higher priority than the other. However the most appropriate statistic is B when we have a set of Gold Standard results we want to give priority too and evaluate (human or machine) generated results against (in terms of predicting the "true" labels), while M tells us how much information there is in the other direction (how much the Gold standard "truth" allows predicting our ratings, how much information about a disease actually flows through to our diagnostic, for example, or to the opinions or ratings of various doctors). In the diagnosis context, B tells us the probability we are making an informed diagnosis (as opposed to random guesses) while M tells us the probability the disease leaves behind markers our test can identify (as opposed to random values).

Reference

Powers, D.M.W (2012). The Problem of Kappa. In Conference of the European Chapter of the Association for Computational Linguistics. ACL. http://aclweb.org/anthology-new/E/E12/E12-1035.pdf