3
$\begingroup$

I'm using R's kappa2 function (irr package) to obtain Cohen's Kappa and the cohen.kappa function (psych package) for the corresponding confidence intervals. In case of significant interrater agreement I usually obtain a p-value ≤ 0.05 and a 95% CI that doesn't include 0. Now, for the first time, the output is a significant p-value, but the 95% CI spans 0. Is there another explanation for this discrepancy, other than my code being erroneous?

> x <- matrix(c(y,z), nrow=49, ncol=2, byrow=FALSE)
> kappa2(x, weight="unweighted", sort.levels="FALSE")

Cohen's Kappa for 2 Raters (Weights: unweighted)
 Subjects = 49 
   Raters = 2 
    Kappa = 0.222 

        z = 2.47 
  p-value = 0.0133 

> cohen.kappa(x)

Call: cohen.kappa1(x = x, w = w, n.obs = n.obs, alpha = alpha, levels = levels)

Cohen Kappa and Weighted Kappa correlation coefficients and confidence boundaries 
                 lower estimate upper
unweighted kappa -0.14     0.22  0.59
weighted kappa   -0.14     0.22  0.59

 Number of subjects = 49 
$\endgroup$
1
  • $\begingroup$ It would be helpful if you included y and z data in your question for a fully reproducible example. $\endgroup$
    – Brian D
    May 10, 2021 at 15:29

2 Answers 2

2
$\begingroup$

There are two ways of computing the standard error of $\kappa$. One is assuming the true value is zero and is appropriate for testing whether $\kappa = 0$, a hypothesis which is seldom of any interest. The other does not assume that and is appropriate for setting confidence intervals. Without a knowledge of the internals of those packages it is impossible to be sure that this is the solution but it seems likely.

The issue is discussed in a paper by Fleiss and colleagues entitled "Large sample standard errors of kappa and weighted kappa"

$\endgroup$
2
  • $\begingroup$ (+1) The psych::cohen.kappa routine uses formula (8) in that article, which seems to be appropriate for computing a confidence interval, and not only test $H_0: \kappa = 0$. $\endgroup$
    – chl
    Nov 17, 2020 at 16:21
  • $\begingroup$ I believe that irr::kappa2 uses formula (9) in that article. $\endgroup$
    – Brian D
    Feb 8, 2021 at 16:17
1
$\begingroup$

If you show the output from str(cohen.kappa(x)), it includes the weighted and unweighted variance values.

You can compare these values to those obtained from irr::kappa2(), which does not report the variance by default. However, based on the kappa2 function code, it appears you can calculate the variance using:

((out$value/out$statistic))^2, where out = kappa2(x, weight="unweighted").

Note that this is typically a different value than that reported by psych::cohen.kappa - the two functions use slightly different equations for kappa - which is why you are noticing the disparity.

Using the computed variance from kappa2, you can see that its CI does not contain 0, which is why the p-value is 'significant' :

.222 + qnorm(c(.025,.975))*(.222/2.47)
[1] 0.04584129 0.39815871
$\endgroup$
2
  • $\begingroup$ Hi, @Brian D Could you please explain the formula used here to calculate the confidence interval for kappa (.222 + qnorm(c(.025,.975))*(.222/2.47)? The formula looks different from what I saw else where (stats.stackexchange.com/questions/208613/…) and not sure whether the two are actually doing the same thing. $\endgroup$
    – Chloe
    May 5, 2021 at 2:26
  • 1
    $\begingroup$ Looking at the source code for irr::kappa2, you can see that the p-value is computed as p.value <- 2 * (1 - pnorm(abs(u))), where u <- value/SEkappa and SEkappa <- sqrt(varkappa), in the above example value = .222 and statistic = u = 2.47. Since the package uses pnorm for the p-value, I use qnorm for the corresponding 95% CI. $\endgroup$
    – Brian D
    May 10, 2021 at 15:39

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.