Are world cup predictions testable? As of today, dozens of soccer world cup predictions exist, some more complex, some more elegant, and most of them predict every nation's "chance" of winning a particular match/ the cup. 
As I am writing a blog post for a newspaper and want to make the statement that these predictions are nice but essentially worthless as they can not be tested, I would like to bulletproof my assumptions and enhance my limited statistical knowledge with this question.
First: Chance. Weirdly, for instance, although Nate Silver explains his model pretty well, he does never explain the actual notion of "chance".  If 538's model "predicts" that Brazil has a 45% chance of winning the tournament, this does indeed mean that if the exact same cup was played a thousand times, and the model would be correct, Brazil would win 450 times on average. And not win in 550 cups, on average. Am I right?  
Second: Evaluation of the predictive power of a model: My even more limited knowledge of data mining, machine learning and predictive modelling tells me that usually, in order to assess the strength of a model, a procedure such as x-validation or validation with a dedicated test set is used. For this to be possible, a test set must exist. In the case of the world cup, this would actually mean that a model such as 538's would have to be applied to either the past 20+ wold cups (not possible because of lacking historical data and extremely tedious) or to countless "instances" of the current world cup (not possible because of.. well yeah). 
Just saying that a model is a good predictor because it correctly predicted (most of the matches of) this world cup 2014 is not strictly valid, is it? In the same vein, when Nate Silver states that SPI worked pretty well for 2010, this actually tells us nothing as it could have been just as well chance, in the sense of a random outcome? So, we have no method of telling whether one of these so prominent world cup predictions are actually good models, given we are not applying these model to the next 100 world cups and evaluating them afterwards? 
What do you mean? 
 A: Yes, World Cup predictions are testable. In addition to the great comments above, here is one way to think about it: 
The 45% probability that Brazil wins (or won, since my answer is years late) is not drawn out of thin air. Instead, it comes out of simulations of the outcomes of individual matches, in which the model predicts wins and losses, with some confidence attached to those predictions as well. Therefore, instead of one single prediction (Brazil wins!), you have predictions of every single match in the tournament, and uncertainty in each of those predictions. 
In this case, you can do a fairly sophisticated check. Of the list of matches in which Silver's model predicts that one team wins with 90% probability, it should be correct 9/10 times. Of matches which it predicts that one team has only a 60% probability of winning, they should be correct 6/10 times. And so on.
My point here is a simple one: instead of having just one prediction, you have a a very large number of them to test, which gives you a simple, accurate way to assess how well the model performed. 
Of course, none of this is new for Silver. He rose to prominence not because he predicted the outcome of the 2008 US federal elections. Many models, both formal and informal, did this. His work was considered impressive because it correctly predicted 49/50 state-level results in the presidential elections, and of every single Senate race. And he also generally made accurate predictions of the margin of victory. 
