I've got a study with two radiologist's reading chest x-rays and want to calculate Cohen's kappa of their agreement. The kappa2 function in the "irr" package and cohen.kappa in "psych" can both give me an answer but don't generate a 95% confidence interval. Is there any good way to do this in R as well?

  • $\begingroup$ Questions solely about how software works are off-topic here, but you may have a real statistical question buried here. You may want to edit your question to clarify the underlying statistical issue. You may find that when you understand the statistical concepts involved, the software-specific elements are self-evident or at least easy to get from the documentation. $\endgroup$ – gung - Reinstate Monica Apr 21 '16 at 14:50
  • 2
    $\begingroup$ The documentation for cohen.kappa in psych suggests that it does, in fact, produce $\alpha/2$ confidence intervals. $\endgroup$ – Timothy Teräväinen Apr 21 '16 at 15:39
  • 1
    $\begingroup$ A simple way is to use bootstrap. Provide your complete code and I can try working out the bootstrap percentile confidence interval. $\endgroup$ – Joe_74 Apr 23 '16 at 10:02

If you have two raters, no missing ratings, and are using the 1960 unweighted kappa, you can calculate the conditional variance using the following formula (see references below):

$$ v = \frac{1-f}{n(1-p_e)^2}\Bigg( p_a(1-p_a)-4(1-\kappa)\Big(\sum_{k=1}^qp_{kk}\hat{\pi}_k-p_a p_e\Big)+4(1-\kappa)^2\Big(\sum_{k=1}^q\sum_{l=1}^qp_{kl}[(p_{Al}+p_{Bl})/2]^2-p_e^2\Big)\Bigg) $$

This variance estimate can then be used to calculate the bounds of a confidence interval.

$$ CI = \kappa \pm \sqrt{v}\times t_{(1-\alpha/2, n-1)}$$


Fleiss, J. L., Nee, J. C. M., & Landis, J. R. (1979). The large sample variance of kappa in the case of different sets of raters. Psychological Bulletin, 86, 974-977.

Gwet, K. L. (2008). Computing inter-rater reliability and its variance in the presence of high agreement. British Journal of Mathematical and Statistical Psychology, 61, 29-48.


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.