How to compare predictive power of sports odds For my MSc Thesis I would like to compare the predictive power of classic bookmakers on the one hand and a betting exchange on the other. I have a lot of data on both types of betting, mostly on soccer matches but I have data on other sports as well. I want to see from the implied probability in the odds if the betting exchange performs better. 
I read about ROC-curve and Area under the curve as a measure but I am not sure if I can use this method in my case? Or do you know a better way see which odds predict the outcome of matches significantly better?
Thanks in advance
 A: The ROC is related to what you're interested in, but not quite it. We're interested in the performance of probability estimates, rather than binary classification thresholds.
In general, methods of evaluating probability estimates are called scoring rules. For every event, you can calculate the implied probability of the event that actually happened, take the log of that probability, and then sum them, ending up with a total score for each oddsmaker.
This gives you a sense of who is better at estimating probabilities overall, but is a single opaque number. For more clarity, let's investigate the two different things that we typically want to estimate: 1) calibration and 2) skill.
Calibration is easier to talk about: it's whether or not I can communicate my belief correctly to probabilities. If I think that team X is 70% likely to win a match, is it the case that across all times I predict a victory with 70% probability that team wins roughly 70% of the time?
We can measure calibration by binning the predictions into the various odds, calculating the actual rate, and then plotting. (You'll also want to do this with a confidence interval; if I only predict 80% twice, and get it right once and wrong once, that doesn't mean much, even though 50% is very different from 80%.)
Skill is how good I am at making right beliefs in the first place. If I only give predictions of "95%" and "5%", and am right 70% of the time in each case, I'm very poorly calibrated (I can't tell the difference between "only somewhat sure" and "very sure.") and would be better off reporting "70%" and "30%." But being right 70% of the time is my skill, and that may be very good (especially compared to chance, which would be right only 50% of the time).
(For further reading, you might be interested in this paper on estimating the calibration and skill of weather predictions.)
