I am writing with questions about a study I am helping with:

Two “raters” will score athletes subjectively from 0 to 3 (3>2>1>0) on each of 10 skill performances. Scores will be summed to a maximum of 30, and the total bracketed into approximately three categories for comparison with regard to knee injury incidence. Reliability/agreement between the two raters’ scores must be determined.

How should the inter-rater reliability testing be set up? Kappa and/or Kendall? Same data for individual skills and total score? How much data is needed for this irr testing?

The lead investigator anticipates highly consistent scores between raters and plans to divide their testing of subjects accordingly, for the main study.

  • $\begingroup$ Kindly narrow down your questions to those not already answered by existing threads on this site. $\endgroup$
    – rolando2
    Sep 7, 2011 at 0:55
  • 1
    $\begingroup$ @rolando2 Could you please provide a link to (at least) one such thread? $\endgroup$
    – whuber
    Sep 7, 2011 at 16:08
  • $\begingroup$ Further to whuber's comment, I outlined the nitty gritty of one part of a study. It is very specific, and there are just a few questions. I looked though other postings. If they do answer the questions, links or references would help. Can someone address these few questions? Thank you. $\endgroup$
    – Jeff
    Sep 7, 2011 at 22:07
  • $\begingroup$ Some existing threads with relevant info: stats.stackexchange.com/questions/7208/… and stats.stackexchange.com/questions/3539/… and stats.stackexchange.com/questions/12415/… $\endgroup$
    – rolando2
    Sep 7, 2011 at 22:28
  • $\begingroup$ I read Stemler, Steven E. (2004). A comparison of consensus, consistency, and measurement approaches to estimating interrater reliability. Practical Assessment, Research & Evaluation, 9(4). PAREonline.net/getvn.asp?v=9&n=4 . $\endgroup$
    – Jeff
    Sep 13, 2011 at 6:40


Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.