Is it valid to report the mean of Cohen's kappa coefficients and the associated confidence intervals?

Question

I am studying the interrater and intrarater reliability of a classification system (seven categories; nominal).

I recruited 16 unique raters to classify 35 items according to the system. The raters classified all items twice (two sessions, one month apart)

I am assessing interrater reliability using Fleiss' kappa. This approach results in 2 values, one per session.

I am assessing intrarater reliability using Cohen's kappa. This approach results in 16 values, one per rater.

So, for each rater, I have the value of Cohen's kappa and the associated confidence interval. I am interested in a summary measure (intrarater reliability) for all 16 raters. Is it valid to report the mean Cohen's kappa coefficient, mean lower bound of the 95% CI, and mean upper bound of the 95% CI?

Thank you.

---

EXAMPLE

Format: Cohen's kappa coefficient (95% CI)

• 0.40 (0.20–0.60)
• 0.10 (–0.10–0.30)
• 0.50 (0.25–0.75)
• 0.45 (0.25–0.65)

Is it valid to report the mean? (i.e., 0.36 [0.15–0.57])

Answer 1

If you have the standard errors of the kappa values and are willing to assume they are normally distributed you could summarise them using techniques from meta-analysis. Each would be inversely weighted by its variance to form the summary and the weights used to form the standard error of the summary and hence to form confidence intervals. Meta-analysis is available in Stata and R, I would not try doing it by hand so I do not give the formulas here.

The CRAN Task View on MetaAnalysis contains more than 100 packages for some variety of meta-analysis. Disclaimer, I maintain the Task View.

Answer 2

It's okay to report the mean of Cohen's kappa. However instead of reporting mean upper/lower confidence limits, report the mean standard error.