outlier detection for subjective rating data which is not consistent among raters In my experiment, I invited 3 experts to evaluate dance performance of 20 subjects. So for each subject, he or she will get 3 scores given by expert individually. For most of the subjects, 3 experts' evaluation are consistent. However, for several subjects, the score given by experts have big difference. For example, for subject 01, expert A gives score 70, expert B give score 50, expert C gives score 30. But for other subjects, the differences are very small. I already used intra-class correlation coefficient to check the consistency among 3 experts' evaluation, it turned out okay, which means the overall evaluation among 3 expert are consistent. Therefore, I want to know is there some statistic methods to detect the subjects whose score is not consistent among 3 experts? I want to remove these subjects as outliers.Thank you in advance.

 A: I don't have an answer, but I do have an advice, and it's the same advice my mother used to give me

The first thing you do, with any sort of data, is you plot it.

In lieu of real data I've had to do with simulations. Hope it fits somewhat.
set.seed(1)

ee <- sort(rbeta(20, 2, 3))
ea <- ee + (rbeta(20, 2, 3) - 0.5)*0.1
eb <- ee + (rbeta(20, 3, 3) - 0.1)*0.1
ec <- ee + (rbeta(20, 4, 2) - 0.1)*0.1

e <- cbind(ea, eb, ec)
e <- e + sample(c(rep(0, 20), 0.2, 0.3), length(e), replace=TRUE)
e <- floor(e / max(e) * 100)

par(mar=c(2, 2, 0.5, 0.5))
matplot(e, type="l", lty=1, lwd=2)
legend("topleft", legend=LETTERS[1:3], col=1:3, lwd=2)


As you can see, the three judges are in broad agreement, but there are some idiosyncrasies. For example:  


*

*Judge A gives on average lower scores than the other two.

*There are a few obvious instances where one of the judges has had a very different impression from the other two. Like with subject 4, 5 and 11, where judge C gives by far the highest grades.

*Less obvious is the case with subject 15. Although the score given by judge A isn't all that high compared to the other two's, it is significant in that it's the only case where judge A gives the highest score of all.
So which of these are outliers? Which of these should be removed? Maybe 6, maybe 12, maybe half, maybe none? It all depends.
Maybe outliers shouldn't be removed completely, but rather given less weight in further analysis? Maybe the distributions should be scaled or transformed in another way? It all depends.
A: An easy way to quantify the consistency in ratings among the raters, on a per-subject basis, is to compute the standard deviation (or range, or mean absolute deviation from the median) of the three ratings.
Whether it makes sense to remove these inconsistently rated subjects as outliers isn't clear; I would have to know more about your analysis and your analytic goal. But I doubt it does.
