Nonparametric bootstrap to construct confidence intervals for Cohen's Kappa Coefficient

Question

I was recently asked to perform nonparametric bootstrap to construct 95% confidence intervals for $\kappa$ using normal approximation, but I'm not sure how to do this. The only data I was given was the table below:
| Clinical trial | |---------------|----------|----------| | Meta analysis | Positive | Negative | | Positive | 13 | 6 | | Negative | 7 | 14 | I believe the $\kappa$ that they are referring to in this case is the Cohen's Kappa Coefficient. Can someone please shed some light on this for me?

Answer 1

Original (unwighted) Cohen's kappa

Cohen's kappa is defined as:

$\kappa = 1 - \dfrac{1-p_e}{1-p_o}$

Where $p_e$ and $p_o$ are the expected (due to chance) and observed agreement, respectively. The expression for $p_e$ assumes that the joint probability distribution of positive and negative factors into a product of the marginal probability distributions for the two observers. So,

$p_e = \dfrac{1}{N^2}\sum\limits_{k} n_{k1}n_{k2} = \dfrac{1}{40^2}(20\times 19 + 20\times 21)$, k $\in$ {positive, negative}

$p_o = \dfrac{a_{11} + a_{22}}{a_{11} + a_{22} + a_{12} + a_{21}}= \dfrac{13 + 14}{13 + 14 + 6 + 7}$, in your example.

Now, to bootstrap you need to sample from your distribution with replacement. One way to do this is to make a vector with 13, 14, 6, and 7 elements each of $a_{11}, a_{22}, a_{12},$ and $a_{21}$, respectively (i.e. the length of the vector is 40). Now you sample, say, 1000 vectors of length 40 from that vector with replacement and each time calculate the kappa as above. The 95% bootstrap confidence interval will be the 2.5 and 97.5 percentile of the calculated kappa values.

Weighted Cohen's Kappa

For a weighted kappa use instead (NOTE: not relevant for your $2\times 2$ example):

$\kappa = 1 - \dfrac{\sum_{i,j}\text{w}_{i,j}O_{i,j}}{\sum_{i,j}\text{w}_{i,j}E_{i,j}}$ for i, j $\in$ {the ordinal categories}.

$O_{i,j}$ is just the observed proportions of $a_{i, j}$ in each bootstrap sample and

$E_{i,j}$ is the product of the marginal distributions, e.g.

$E_{1, 1} = \dfrac{(13 + 6)\times (13 + 7)}{40}$, in your example.

Finally, the weight matrix can be chosen arbitrarily but are usually derived as linear or quadratic by:

$w_{i,j} = \dfrac{(i-j)^x}{(N-1)^x}$, with x=1 for linear kappa and x=2 for quadratic kappa.

The linear kappa gives a linearly larger penalty for ordinal categories when the distance of the disagreement increases, and the quadratic kappa gives a smaller penalty for short distances and a larger penalty to large distances, relative to the linear weights.