2
$\begingroup$

In a study on intra-rater agreement with 8 raters, I have computed Cohen's Kappa for each of the raters and I estimated the standard error of each of the kappa values using the Jackknife method.

To get an estimate of the overall intra-rater agreement, I took the mean of all kappa values. So far so good, but what is the correct way to combine the standard errors? I want to combine these standard errors, so I can compute a 95% CI for the mean kappa.

I first thought that I could combine the standard errors by using the formula for pooled variance (wikipedia:pooled variance, function in R), with the kappa values as the means, the squared standard errors as the variances and the number of observations as N. However, I am not sure whether this is entirely correct. Can I replace the mean in the formula by the estimated kappa? If this is incorrect, can someone explain it to me from a mathematical point of view and a statistical point of view? Furthermore, I got upper confidence limits > 1 and hence, a value that is impossible for kappa.

$\endgroup$
1
$\begingroup$

There is a version of kappa for the case of multiple raters which is usually attributed to Conger. A quick search with your favourite statistical package should find how to compute it. This is likely to be more productive than trying to average values of pair-wise kappa and then compute a variance for the average.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.