Comparing Group and Expert Result Say I am predicting swimming times. I get a group of 20 to make predictions on the fastest to slowest swimmers from 1 to 6. I then ask a single expert in swimming to make the same predictions. Is there a way to test statistical significance between the group's predictions versus the expert's?
"significant" would mean if I were to select a random person from the group there's a very low percent he would predict in the same order as the expert.
ex.
We rank 1 to 6 the swimmers who received the highest to lowest average predictions (10 people predict swimmer 1 to get first, 10 predict swimmer 1 to get second, no one predicts swimmer to get other; his average is then 1.5, average of the predictions. We do this for every swimmer and rank the first 6 according to average). Now we have our top 6 from the group, how would I determine significance (as defined above) of an expert's prediction on the same swimmers.
 A: It depends on the question. Interrater agreement has been studied extensively, so there is not one single way to do this. I'll pose a question to you: if rater 1 predicts Swimmer A will take 6th place and Swimmer B will take 1st place whereas rater 2 predicts Swimmer A will take 1st place and swimmer B will take 6th place, would you or should you consider that a worse ranking than a similar setting where the disagreement is over 1st versus 2nd place? I can say with some confidence it would be er
I would think that a weighted kappa is the best approach to evaluating rater agreement for ranks. The choice of weight is determined by the problem. A higher weight considers adjacent ranks to be "ball park" close so that the second scenario of my example is considered an "okay" agreement but the first scenario is considered "awful". Unweighted kappas classify any ranking disagreement as a complete miss, regardless of how close it was. Confidence intervals and hypothesis tests are available for kappas using standard software. However, it would require that you actually produce an experiment for this question, and simulate examples from many types of races/events.
