Say I am predicting swimming times. I get a group of 20 to make predictions on the fastest to slowest swimmers from 1 to 6. I then ask a single expert in swimming to make the same predictions. Is there a way to test statistical significance between the group's predictions versus the expert's?
"significant" would mean if I were to select a random person from the group there's a very low percent he would predict in the same order as the expert.

We rank 1 to 6 the swimmers who received the highest to lowest average predictions (10 people predict swimmer 1 to get first, 10 predict swimmer 1 to get second, no one predicts swimmer to get other; his average is then 1.5, average of the predictions. We do this for every swimmer and rank the first 6 according to average). Now we have our top 6 from the group, how would I determine significance (as defined above) of an expert's prediction on the same swimmers.

  • $\begingroup$ Different in what sense? Can you give an example of a "significant" and a "nonsignificant" difference? That might help clarify your intention here $\endgroup$ – shadowtalker Aug 12 '16 at 4:25
  • $\begingroup$ Different in the sense that if I were to choose 1 person from my group randomly what are the chances his prediction would match the expert's prediction given the distribution of the entire group. I guess "significant" would mean if I were to select a random person from the group there's a very low percent he would predict in the same order as the expert. $\endgroup$ – Allen Lu Aug 12 '16 at 8:03
  • $\begingroup$ Thanks for clarifying. You should edit that into the body of the question itself; comments are not searchable, linkable, or archived in the same way that question text is. $\endgroup$ – shadowtalker Aug 12 '16 at 14:20
  • $\begingroup$ "Significance" does not seem to be an applicable concept. You seem to be asking for a way of measuring or assessing the discrepancy between the actual outcome and any particular prediction of it. Since there are an abundance of solutions, what additional information can you supply to narrow them down? Could you explain how you plan to use this measure of predictive accuracy, for instance? $\endgroup$ – whuber Aug 12 '16 at 18:39
  • $\begingroup$ The goal of the problem is to find if there is a difference between the amateur group and the expert. There is no actual outcome yet, we are just checking if there is a difference. We want a score that can tell us how different an expert's result is compared with the amateur group's predictions. In a sense it is a classification problem, but I'm not sure how to go about it, since rankings are dependent of one another. $\endgroup$ – Allen Lu Aug 12 '16 at 19:43

It depends on the question. Interrater agreement has been studied extensively, so there is not one single way to do this. I'll pose a question to you: if rater 1 predicts Swimmer A will take 6th place and Swimmer B will take 1st place whereas rater 2 predicts Swimmer A will take 1st place and swimmer B will take 6th place, would you or should you consider that a worse ranking than a similar setting where the disagreement is over 1st versus 2nd place? I can say with some confidence it would be er

I would think that a weighted kappa is the best approach to evaluating rater agreement for ranks. The choice of weight is determined by the problem. A higher weight considers adjacent ranks to be "ball park" close so that the second scenario of my example is considered an "okay" agreement but the first scenario is considered "awful". Unweighted kappas classify any ranking disagreement as a complete miss, regardless of how close it was. Confidence intervals and hypothesis tests are available for kappas using standard software. However, it would require that you actually produce an experiment for this question, and simulate examples from many types of races/events.

  • $\begingroup$ Thanks for the answer. It got me looking more into comparing ordinal results. One thing I found that looked interesting was Kendell's tau. I understand both of them deal with the problem, but I'm confused on the exact difference and when to use which. $\endgroup$ – Allen Lu Aug 18 '16 at 1:17

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.