Significant Kappa coefficient with 95% CI spanning 0?

I'm using R's kappa2 function (irr package) to obtain Cohen's Kappa and the cohen.kappa function (psych package) for the corresponding confidence intervals. In case of significant interrater agreement I usually obtain a p-value ≤ 0.05 and a 95% CI that doesn't include 0. Now, for the first time, the output is a significant p-value, but the 95% CI spans 0. Is there another explanation for this discrepancy, other than my code being erroneous?

> x <- matrix(c(y,z), nrow=49, ncol=2, byrow=FALSE)
> kappa2(x, weight="unweighted", sort.levels="FALSE")

Cohen's Kappa for 2 Raters (Weights: unweighted)
Subjects = 49
Raters = 2
Kappa = 0.222

z = 2.47
p-value = 0.0133

> cohen.kappa(x)

Call: cohen.kappa1(x = x, w = w, n.obs = n.obs, alpha = alpha, levels = levels)

Cohen Kappa and Weighted Kappa correlation coefficients and confidence boundaries
lower estimate upper
unweighted kappa -0.14     0.22  0.59
weighted kappa   -0.14     0.22  0.59

Number of subjects = 49

• It would be helpful if you included y and z data in your question for a fully reproducible example. May 10, 2021 at 15:29

There are two ways of computing the standard error of $$\kappa$$. One is assuming the true value is zero and is appropriate for testing whether $$\kappa = 0$$, a hypothesis which is seldom of any interest. The other does not assume that and is appropriate for setting confidence intervals. Without a knowledge of the internals of those packages it is impossible to be sure that this is the solution but it seems likely.

The issue is discussed in a paper by Fleiss and colleagues entitled "Large sample standard errors of kappa and weighted kappa"

• (+1) The psych::cohen.kappa routine uses formula (8) in that article, which seems to be appropriate for computing a confidence interval, and not only test $H_0: \kappa = 0$.
– chl
Nov 17, 2020 at 16:21
• I believe that irr::kappa2 uses formula (9) in that article. Feb 8, 2021 at 16:17

If you show the output from str(cohen.kappa(x)), it includes the weighted and unweighted variance values.

You can compare these values to those obtained from irr::kappa2(), which does not report the variance by default. However, based on the kappa2 function code, it appears you can calculate the variance using:

((out$value/out$statistic))^2, where out = kappa2(x, weight="unweighted").

Note that this is typically a different value than that reported by psych::cohen.kappa - the two functions use slightly different equations for kappa - which is why you are noticing the disparity.

Using the computed variance from kappa2, you can see that its CI does not contain 0, which is why the p-value is 'significant' :

.222 + qnorm(c(.025,.975))*(.222/2.47)
[1] 0.04584129 0.39815871

• Hi, @Brian D Could you please explain the formula used here to calculate the confidence interval for kappa (.222 + qnorm(c(.025,.975))*(.222/2.47)? The formula looks different from what I saw else where (stats.stackexchange.com/questions/208613/…) and not sure whether the two are actually doing the same thing. May 5, 2021 at 2:26
• Looking at the source code for irr::kappa2, you can see that the p-value is computed as p.value <- 2 * (1 - pnorm(abs(u))), where u <- value/SEkappa and SEkappa <- sqrt(varkappa), in the above example value = .222 and statistic = u = 2.47. Since the package uses pnorm for the p-value, I use qnorm for the corresponding 95% CI. May 10, 2021 at 15:39