5
$\begingroup$

As of today, dozens of soccer world cup predictions exist, some more complex, some more elegant, and most of them predict every nation's "chance" of winning a particular match/ the cup.

As I am writing a blog post for a newspaper and want to make the statement that these predictions are nice but essentially worthless as they can not be tested, I would like to bulletproof my assumptions and enhance my limited statistical knowledge with this question.

First: Chance. Weirdly, for instance, although Nate Silver explains his model pretty well, he does never explain the actual notion of "chance". If 538's model "predicts" that Brazil has a 45% chance of winning the tournament, this does indeed mean that if the exact same cup was played a thousand times, and the model would be correct, Brazil would win 450 times on average. And not win in 550 cups, on average. Am I right?

Second: Evaluation of the predictive power of a model: My even more limited knowledge of data mining, machine learning and predictive modelling tells me that usually, in order to assess the strength of a model, a procedure such as x-validation or validation with a dedicated test set is used. For this to be possible, a test set must exist. In the case of the world cup, this would actually mean that a model such as 538's would have to be applied to either the past 20+ wold cups (not possible because of lacking historical data and extremely tedious) or to countless "instances" of the current world cup (not possible because of.. well yeah). Just saying that a model is a good predictor because it correctly predicted (most of the matches of) this world cup 2014 is not strictly valid, is it? In the same vein, when Nate Silver states that SPI worked pretty well for 2010, this actually tells us nothing as it could have been just as well chance, in the sense of a random outcome? So, we have no method of telling whether one of these so prominent world cup predictions are actually good models, given we are not applying these model to the next 100 world cups and evaluating them afterwards?

What do you mean?

$\endgroup$
11
  • $\begingroup$ When you say "could be chance" about 2010, what do you think is the likelihood that most of the predictions would be correct by chance? $\endgroup$
    – Glen_b
    Jun 16 '14 at 9:51
  • $\begingroup$ A very low likelihood, I would say. Just as low as with every other correct tournament prediction. But does that single outcome validate/verify the model? $\endgroup$
    – grssnbchr
    Jun 16 '14 at 10:03
  • 1
    $\begingroup$ You're left with two possibilities: either a very low probability event occurred with a model that doesn't actually have good predictive ability generally, or the model really did better than chance. If that probability was very low, is the first explanation tenable? (If the model was calibrated on the same data as was being predicted, we should not be impressed, but if the predictions were made on the basis of a model calibrated on other data than he was predicting, it's much more impressive). [Silver understands this already, of course, so I doubt that's an issue we will need to worry about] $\endgroup$
    – Glen_b
    Jun 16 '14 at 10:15
  • 3
    $\begingroup$ Silver's methods have a Bayesian foundation. A natural way to assess the quality of predictions from that point of view is through wagers on the outcomes: how well would somebody betting based on the predictions end up? The worth of that seems eminently testable to me--and should be readily understandable by almost any sports blog reader. $\endgroup$
    – whuber
    Jun 16 '14 at 14:21
  • 2
    $\begingroup$ I'm not exactly clear, but you seem to be setting up a straw man with the way your phrased that. I don't see why you'd insist on a given set of outputs as the thing that's "unfalsifiable" - that would not be the the thing that's falsified for a scientific theory, for example. A theory could be falsifiable on the basis of a test: predict a set of outcomes (that you don't see when constructing the predictions) and then compare the outcomes with the predictions. In the case of statistical predictions, of course, the way to evaluate them has been discussed clearly enough by Silver before. $\endgroup$
    – Glen_b
    Jun 17 '14 at 8:56
1
$\begingroup$

Yes, World Cup predictions are testable. In addition to the great comments above, here is one way to think about it:

The 45% probability that Brazil wins (or won, since my answer is years late) is not drawn out of thin air. Instead, it comes out of simulations of the outcomes of individual matches, in which the model predicts wins and losses, with some confidence attached to those predictions as well. Therefore, instead of one single prediction (Brazil wins!), you have predictions of every single match in the tournament, and uncertainty in each of those predictions.

In this case, you can do a fairly sophisticated check. Of the list of matches in which Silver's model predicts that one team wins with 90% probability, it should be correct 9/10 times. Of matches which it predicts that one team has only a 60% probability of winning, they should be correct 6/10 times. And so on.

My point here is a simple one: instead of having just one prediction, you have a a very large number of them to test, which gives you a simple, accurate way to assess how well the model performed.

Of course, none of this is new for Silver. He rose to prominence not because he predicted the outcome of the 2008 US federal elections. Many models, both formal and informal, did this. His work was considered impressive because it correctly predicted 49/50 state-level results in the presidential elections, and of every single Senate race. And he also generally made accurate predictions of the margin of victory.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.