I am studying the interrater and intrarater reliability of a classification system (seven categories; nominal).
I recruited 16 unique raters to classify 35 items according to the system. The raters classified all items twice (two sessions, one month apart)
I am assessing interrater reliability using Fleiss' kappa. This approach results in 2 values, one per session.
I am assessing intrarater reliability using Cohen's kappa. This approach results in 16 values, one per rater.
So, for each rater, I have the value of Cohen's kappa and the associated confidence interval. I am interested in a summary measure (intrarater reliability) for all 16 raters. Is it valid to report the mean Cohen's kappa coefficient, mean lower bound of the 95% CI, and mean upper bound of the 95% CI?
Format: Cohen's kappa coefficient (95% CI)
- 0.40 (0.20–0.60)
- 0.10 (–0.10–0.30)
- 0.50 (0.25–0.75)
- 0.45 (0.25–0.65)
Is it valid to report the mean? (i.e., 0.36 [0.15–0.57])