As of today, dozens of soccer world cup predictions exist, some more complex, some more elegant, and most of them predict every nation's "chance" of winning a particular match/ the cup.
As I am writing a blog post for a newspaper and want to make the statement that these predictions are nice but essentially worthless as they can not be tested, I would like to bulletproof my assumptions and enhance my limited statistical knowledge with this question.
First: Chance. Weirdly, for instance, although Nate Silver explains his model pretty well, he does never explain the actual notion of "chance". If 538's model "predicts" that Brazil has a 45% chance of winning the tournament, this does indeed mean that if the exact same cup was played a thousand times, and the model would be correct, Brazil would win 450 times on average. And not win in 550 cups, on average. Am I right?
Second: Evaluation of the predictive power of a model: My even more limited knowledge of data mining, machine learning and predictive modelling tells me that usually, in order to assess the strength of a model, a procedure such as x-validation or validation with a dedicated test set is used. For this to be possible, a test set must exist. In the case of the world cup, this would actually mean that a model such as 538's would have to be applied to either the past 20+ wold cups (not possible because of lacking historical data and extremely tedious) or to countless "instances" of the current world cup (not possible because of.. well yeah). Just saying that a model is a good predictor because it correctly predicted (most of the matches of) this world cup 2014 is not strictly valid, is it? In the same vein, when Nate Silver states that SPI worked pretty well for 2010, this actually tells us nothing as it could have been just as well chance, in the sense of a random outcome? So, we have no method of telling whether one of these so prominent world cup predictions are actually good models, given we are not applying these model to the next 100 world cups and evaluating them afterwards?
What do you mean?