Calculating Cohen's Kappa in SPSS for a systematic review

Question

I'm sure there's a simple answer to this but I haven't been able to find it yet. All the explanations I've found to calculate Cohen's Kappa in SPSS use data that is formatted with cases that are listed as rows (see image below for example). Then analysing the data as a cross tabulation.

My problem is that for a systematic review, this way of formatting data is very inconvenient (I would have to list more than 1500 articles as cases). Is there a way of calculating Cohen's Kappa when the data is presented as a square matrix (see image below)? I know how to calculate the statistic manually but my problem is that SPSS generates the p-value which I do not know how to manually calculate.

Answer 1

You can use 'weight cases' in SPSS to tell SPSS to treat that row in the data as repeating N times.

So you make four rows (inc/inc, inc/exc, exc/inc and exc/exc) and add the number in each of these categories, then you can analyze in the usual ways.

Answer 2

You can use the counts from your confusion matrix to simulate data in the long format that SPSS expects.

As a very simple example, imagine your four cells of the confusion matrix were 1, 2, 3, 4 (instead of 100, 20, 40, 100).

Create 1 example of include-include, 2 examples of include-exclude, 3 examples of exclude-include, and 4 examples of exclude-exclude:

RaterA	RaterB
Include	Include
Include	Exclude
Include	Exclude
Exclude	Include
Exclude	Include
Exclude	Include
Exclude	Exclude
Exclude	Exclude
Exclude	Exclude
Exclude	Exclude

Update:

Here is the formula for the two-rater unweighted Cohen's kappa when there is no missing ratings and the ratings are organized in a contingency table.

$$\hat{\kappa} = \frac{p_a - p_e}{1 - p_e}$$

$$p_a = \sum_{k=1}^q p_{kk}$$

$$p_e = \sum_{k=1}^q p_{k+} p_{+k}$$

Here is the formula for the variance of the two-rater unweighted Cohen's kappa assuming the same.

$$v(\hat{\kappa}) = \frac{1-f}{n(1-p_e)^2}\left(p_a(1-p_a)-4(1-\hat{\kappa}\left(\sum_{k=1}^qp_{kk}\hat{\pi}_k - p_a p_e\right) + 4(1-\hat{\kappa})^2\left(\sum_{k=1}^q\sum_{l=1}^qp_{kl}((p_{Al}+p_{Bk})/2)^2 - p_e^2\right)\right)$$

From this variance estimate, you can compute the standard error and calculate a $p$-value and confidence interval.

See Gwet (2014) page 142 for more information.