2
$\begingroup$

I've spent some time looking through literature about sample size calculation for Cohen's kappa, and in several studies there are stated that increasing the number of raters, reduce the number of subjects required to get the same power (which I think is logical) when looking at inter-rater reliability by use of Kappa statistics. But there is, as far as I can see, no specific calculation or reference for the statement. In this link there is calculation for 2 raters.

  • Is anyone familiar with similar calculation for several raters?
  • Other factors that would affect the number of subjects required?

I will (probably) have 5 categories of nominal data. There might be combined findings. There will be 3 raters.

I found this article saying something about sample size and several raters: Sim, J. and Wright, C. C. (2005) ‘Interpretation, and Sample Size Requirements The Kappa Statistic in Reliability Studies: Use’, Journal of the American Physical Therapy Association, 85, pp. 257–268.

When seeking to optimize sample size, the investigator needs to choose the appropriate balance between the number of raters examining each subject and the number of subjects. In some instances, it is more practical to increase the number of raters rather than increase the number of subjects. However, according to Shoukri, when seeking to detect a kappa of .40 or greater on a dichotomous variable, it is not advantageous to use more than 3 raters per subject—it can be shown that for a fixed number of observations, increasing the number of raters beyond 3 has little effect on the power of hypothesis tests or the width of confidence intervals. Therefore, increasing the number of subjects is the more effective strategy for maximizing power.

|cite|improve this question
$\endgroup$
  • $\begingroup$ Power for kappa is a bit weird. We usually aren't looking for statistically significant values of kappa, because the kappa might be highly significant, but still indicate low agreement. $\endgroup$ – Jeremy Miles Nov 22 '15 at 22:09
  • 2
    $\begingroup$ You are right. But still there are requirements for sample size to ensure validity of results. By a relative error of i.e. 30% and a pa-pe of i.e. 0,6, the table here agreestat.com/blog_irr/sample_size_determination.html suggests 31 subjects. What would this number be for 3 raters? $\endgroup$ – Siv Nov 23 '15 at 14:23
  • $\begingroup$ I see. They are talking about getting an acceptable standard error. $\endgroup$ – Jeremy Miles Nov 23 '15 at 14:41
3
$\begingroup$

You can use R package "KappaSize" for sample size calculation or power analysis for raters more than two and categories more than two. The function can be "Power5Cats"

|cite|improve this answer
$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.