For 5 raters, I have calculated intra-rater Cohen's Kappa statistics for the test-retest nominal ratings, intra-rater Kendall's tau-b statistics for the test-retest ordinal ratings, and intra-rater ICCs for the test-retest scale ratings. Is it valid to calculate overall mean intra-rater reliability coefficients (Kappa, Kendall's tau-b, ICC, as appropriate) across all raters for each of these variables? If so, can I simply average the individual raters' reliability coefficients for overall means?

  • $\begingroup$ The average of such reliability coefficients is, just that, the average of such reliability coefficients. Why would it be interesting or useful? Best to design an analysis of all kinds of variation, between and within raters. $\endgroup$ – Nick Cox Jun 4 '13 at 18:18

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Browse other questions tagged or ask your own question.