# Can you test if one judge gives significantly different scores from other judges?

If I had a small number of observations (about 5) in a sample and the population standard deviation was unknown, what test could be done to see if the value of one of the observations was significantly different from the mean? An example of this would be to test if one judge out of 5 judges in a competition was giving a score significantly different from other judges.

• The answer is yes, but if you're using the same data to test it as you did to identify the judge you want to test, your p-values will be meaningless (unless they take account of that fact). That is, if you say "man, that judge is harsh" (say), then "let's test if he's scoring differently", you can't test the data that made you notice him. – Glen_b Jun 10 '14 at 0:35
• Is your 'judge giving a different score' just an example, or is it the real question? There are methods for assessing inter-rater agreement. – gung Jun 10 '14 at 0:37
• There's an example here of a t-test with $n_1=3$ vs $n_2=1$, as an indication that (with some uncheckable assumptions) such things are possible -- but @gung's note about inter-rater agreement may be nearer to what you need (with the same caution about using the data that made you want to test). But it also seems to me that a single judge in a competition would presumably give a string of scores, not a single score, so this doesn't sound like an $n=1$ problem. Can you clarify? – Glen_b Jun 10 '14 at 0:41
• @gung,the judge example is the real question. I am not trying to resolve a particular situation with a particular judge at this time, but I would like to be prepared just in case the occasion does occur. – eglinker Jun 10 '14 at 1:09
• Will you really need a significance test? As @Glen_b notes, if you notice one judge giving different scores & decide to test that judge, the p-values will be meaningless. Will there be just one score, or will there be many scores you can use (eg, ratings of ice skaters by different judges)? You might also be interested in multivariate outlier detection. – gung Jun 10 '14 at 1:19