# Are world cup predictions testable?

As of today, dozens of soccer world cup predictions exist, some more complex, some more elegant, and most of them predict every nation's "chance" of winning a particular match/ the cup.

As I am writing a blog post for a newspaper and want to make the statement that these predictions are nice but essentially worthless as they can not be tested, I would like to bulletproof my assumptions and enhance my limited statistical knowledge with this question.

First: Chance. Weirdly, for instance, although Nate Silver explains his model pretty well, he does never explain the actual notion of "chance". If 538's model "predicts" that Brazil has a 45% chance of winning the tournament, this does indeed mean that if the exact same cup was played a thousand times, and the model would be correct, Brazil would win 450 times on average. And not win in 550 cups, on average. Am I right?

Second: Evaluation of the predictive power of a model: My even more limited knowledge of data mining, machine learning and predictive modelling tells me that usually, in order to assess the strength of a model, a procedure such as x-validation or validation with a dedicated test set is used. For this to be possible, a test set must exist. In the case of the world cup, this would actually mean that a model such as 538's would have to be applied to either the past 20+ wold cups (not possible because of lacking historical data and extremely tedious) or to countless "instances" of the current world cup (not possible because of.. well yeah). Just saying that a model is a good predictor because it correctly predicted (most of the matches of) this world cup 2014 is not strictly valid, is it? In the same vein, when Nate Silver states that SPI worked pretty well for 2010, this actually tells us nothing as it could have been just as well chance, in the sense of a random outcome? So, we have no method of telling whether one of these so prominent world cup predictions are actually good models, given we are not applying these model to the next 100 world cups and evaluating them afterwards?

What do you mean?

• When you say "could be chance" about 2010, what do you think is the likelihood that most of the predictions would be correct by chance? Jun 16 '14 at 9:51
• A very low likelihood, I would say. Just as low as with every other correct tournament prediction. But does that single outcome validate/verify the model? Jun 16 '14 at 10:03
• You're left with two possibilities: either a very low probability event occurred with a model that doesn't actually have good predictive ability generally, or the model really did better than chance. If that probability was very low, is the first explanation tenable? (If the model was calibrated on the same data as was being predicted, we should not be impressed, but if the predictions were made on the basis of a model calibrated on other data than he was predicting, it's much more impressive). [Silver understands this already, of course, so I doubt that's an issue we will need to worry about] Jun 16 '14 at 10:15
• Silver's methods have a Bayesian foundation. A natural way to assess the quality of predictions from that point of view is through wagers on the outcomes: how well would somebody betting based on the predictions end up? The worth of that seems eminently testable to me--and should be readily understandable by almost any sports blog reader.
– whuber
Jun 16 '14 at 14:21
• I'm not exactly clear, but you seem to be setting up a straw man with the way your phrased that. I don't see why you'd insist on a given set of outputs as the thing that's "unfalsifiable" - that would not be the the thing that's falsified for a scientific theory, for example. A theory could be falsifiable on the basis of a test: predict a set of outcomes (that you don't see when constructing the predictions) and then compare the outcomes with the predictions. In the case of statistical predictions, of course, the way to evaluate them has been discussed clearly enough by Silver before. Jun 17 '14 at 8:56