# Calculating Cohen's Kappa in SPSS for a systematic review

I'm sure there's a simple answer to this but I haven't been able to find it yet. All the explanations I've found to calculate Cohen's Kappa in SPSS use data that is formatted with cases that are listed as rows (see image below for example). Then analysing the data as a cross tabulation.

My problem is that for a systematic review, this way of formatting data is very inconvenient (I would have to list more than 1500 articles as cases). Is there a way of calculating Cohen's Kappa when the data is presented as a square matrix (see image below)? I know how to calculate the statistic manually but my problem is that SPSS generates the p-value which I do not know how to manually calculate.

You can use 'weight cases' in SPSS to tell SPSS to treat that row in the data as repeating N times.

So you make four rows (inc/inc, inc/exc, exc/inc and exc/exc) and add the number in each of these categories, then you can analyze in the usual ways.

• Thanks for your solution, Jeremy! That's exactly what I was looking for.
– Iris
May 12, 2022 at 6:39

You can use the counts from your confusion matrix to simulate data in the long format that SPSS expects.

As a very simple example, imagine your four cells of the confusion matrix were 1, 2, 3, 4 (instead of 100, 20, 40, 100).

Create 1 example of include-include, 2 examples of include-exclude, 3 examples of exclude-include, and 4 examples of exclude-exclude:

RaterA RaterB
Include Include
Include Exclude
Include Exclude
Exclude Include
Exclude Include
Exclude Include
Exclude Exclude
Exclude Exclude
Exclude Exclude
Exclude Exclude

Update:

Here is the formula for the two-rater unweighted Cohen's kappa when there is no missing ratings and the ratings are organized in a contingency table.

$$\hat{\kappa} = \frac{p_a - p_e}{1 - p_e}$$

$$p_a = \sum_{k=1}^q p_{kk}$$

$$p_e = \sum_{k=1}^q p_{k+} p_{+k}$$

Here is the formula for the variance of the two-rater unweighted Cohen's kappa assuming the same.

$$v(\hat{\kappa}) = \frac{1-f}{n(1-p_e)^2}\left(p_a(1-p_a)-4(1-\hat{\kappa}\left(\sum_{k=1}^qp_{kk}\hat{\pi}_k - p_a p_e\right) + 4(1-\hat{\kappa})^2\left(\sum_{k=1}^q\sum_{l=1}^qp_{kl}((p_{Al}+p_{Bk})/2)^2 - p_e^2\right)\right)$$

From this variance estimate, you can compute the standard error and calculate a $$p$$-value and confidence interval.

• I could share the formula for calculating $p$ for you, but I suspect this isn't what you're looking for if you use SPSS. As an alternative that would still be easy, check out www.agreestat360.com for Gwet's web-based calculator. May 9, 2022 at 21:17