# How to compare two Cohen's Kappa statistics based on the same sample of subjects but different raters

Suppose I have two assessment methods A and B for assessing depression. I first ask two raters (Raters 1 and 2) to use Method A to assess 50 patients. I compute Cohen's Kappa based on the ratings (assuming the ratings are ordinal). I then ask two different raters (Raters 3 and 4) to use Method B to assess the same 50 patients (again, let's assume the ratings are ordinal). I compute Cohen's Kappa based on these two new raters' ratings.

I wonder if there is a way to compare these two kappa statistics -- that is, whether they are statistically different.

## 1 Answer

Your best bet would likely be to construct confidence intervals for each kappa point estimate and see if they overlap; if not, then the two estimates are significantly different from one another. Many functions for calculating kappa will include a confidence interval if you ask for it. For instance, in R, you can use one of these functions to generate Cohen's or Conger's kappa with ordinal weights.