I have data resulting from 3 (fixed) judges observing 15 medical students performing physical exam maneuvers. I'm trying to decide the correct correlation statistic to use. The results can be: done and done correctly (1), attempted (2), or did not attempt (3). I initially thought to use ICC, but I got really screwy results when there was perfect or almost perfect agreement. I think because the data are not normal. Incidentally I am using Stata.

  • 1
    $\begingroup$ I may be incorrect, but I thought one could only use Kappa if there were 2 judges, not more. $\endgroup$ – LDavis Oct 6 '16 at 22:40
  • 2
    $\begingroup$ Agreement is not the same as correlation. It looks like you need measures of agreement, not measures of correlation $\endgroup$ – kjetil b halvorsen Mar 8 '19 at 10:26

The appropriate measure here is a measure of agreement since the rating is a categorical variable. For three raters the usual technique was discussed by Fleiss in a paper entitled "Measuring nominal scale agreement among many raters" available here. The method is quite general and does not assume the same raters throughout.

The issue of what to do when there is near perfect agreement is complex as it depends on the exact data-set but if the problem is that the raters tend to only use one category it is useful to report percent agreement alngside kappa.


I understand that your goal is to compute correlation between different judges to measure how consistent are results.

The first point should be decide what kind of data you have:

  • Categorical data, if there is not an order in your 3 results (correctly is better than attempted, but is attempted better than not attempted?).
  • Ordered data: correctly better than attempted better than not attempted.
  • Quantitative data: Although your data is not actually quantitative, it could be treated as quantitative if there were and order and the "distance" between correctly and attempted were the same as between attempted and not attempted.

I don't know a usual measure of correlation for categorical data in 3 levels, although for 2 levels there is odds ratio and related statistics.

For ordered data - which seems likely - you can use a non parametric measure of rank correlation as Spearman's rank correlation coefficient.

If your data can be seen as quantitative Spearman's would be also fine, but it's more common to use Pearson's correlation coefficient.

Beware that correlation coefficients are computed on pairs of variables; therefore, you will have a correlation coefficient for each pair of judges - most statistical packages have an order to compute it for all pairs in a group and present them as a correlation matrix. ICC does work with several groups at once, although I doubt if it actually can fit your needs.

Furthermore, if you want to test effect of judge and student in results you could use Kruskal Wallis test, although it needs that your data can be seen as quantitative.

  • $\begingroup$ You really need a measure of Agreement, not of correlation. See the answer by @mdewey. $\endgroup$ – kjetil b halvorsen Nov 12 '19 at 1:08

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.