I have data resulting from 3 (fixed) judges observing 15 medical students performing physical exam maneuvers. I'm trying to decide the correct correlation statistic to use. The results can be: done and done correctly (1), attempted (2), or did not attempt (3). I initially thought to use ICC, but I got really screwy results when there was perfect or almost perfect agreement. I think because the data are not normal. Incidentally I am using Stata.
The appropriate measure here is a measure of agreement since the rating is a categorical variable. For three raters the usual technique was discussed by Fleiss in a paper entitled "Measuring nominal scale agreement among many raters" available here. The method is quite general and does not assume the same raters throughout.
The issue of what to do when there is near perfect agreement is complex as it depends on the exact data-set but if the problem is that the raters tend to only use one category it is useful to report percent agreement alngside kappa.
I understand that your goal is to compute correlation between different judges to measure how consistent are results.
The first point should be decide what kind of data you have:
- Categorical data, if there is not an order in your 3 results (correctly is better than attempted, but is attempted better than not attempted?).
- Ordered data: correctly better than attempted better than not attempted.
- Quantitative data: Although your data is not actually quantitative, it could be treated as quantitative if there were and order and the "distance" between correctly and attempted were the same as between attempted and not attempted.
I don't know a usual measure of correlation for categorical data in 3 levels, although for 2 levels there is odds ratio and related statistics.
For ordered data - which seems likely - you can use a non parametric measure of rank correlation as Spearman's rank correlation coefficient.
If your data can be seen as quantitative Spearman's would be also fine, but it's more common to use Pearson's correlation coefficient.
Beware that correlation coefficients are computed on pairs of variables; therefore, you will have a correlation coefficient for each pair of judges - most statistical packages have an order to compute it for all pairs in a group and present them as a correlation matrix. ICC does work with several groups at once, although I doubt if it actually can fit your needs.
Furthermore, if you want to test effect of judge and student in results you could use Kruskal Wallis test, although it needs that your data can be seen as quantitative.