3
$\begingroup$

In a recent post on kdnuggets.com the author looks back at their predictions for the FIFA World Cup 2018 and states what can be learned from the results.

They have trained an ensemble model based on various ranking systems and predictions (like FifeThirtyEight or Gracenote) and predicted Germany to win the tournament. As it turned out, this has not proven to be true.

Randomness in human behavior (cannot be modelled) is stated to be one of the reasons for the model performance.

How can this be randomness in a statistical sense? I do not see a reason to include randomness here. I would have just said that something very unlikely happened and that this does not need to be an indicator for bad modelling. Is it because they ignore the difference between the actual outcome and the true class probability? Here is an example:

The predicted class probability of their model:

  • GER winning: 0.24
  • FRA winning: 0.12

The unknown true class probability (that they tried to model):

  • GER winning: 0.19
  • FRA winning: 0.17

The actual class probability:

  • GER winning: 0.00
  • FRA winning: 1.00
$\endgroup$

1 Answer 1

2
$\begingroup$

Your exact point is made in the initial chapters of Superforecasting: The Art and Science of Prediction by Tetlock and Gardner. The authors discuss the important role of how to assess probabilistic forecasts.

If a weather forecaster forecasted a 80% chance of rain, and the day turned out to be sunny, does that mean that the forecast is bad? No, it just means that we need to assess whether it rains on 80% of similar days to this one. If it rains on 0%, or on 100%, in a large sample, then yes, the forecast is systematically wrong. (It turns out that weather forecasters are more likely to forecast a higher chance of rain than their models output, because people will remember low-forecasted-probability-rain better than low-forecasted-probability-sunshine.)

Tetlock and Gardner discuss and recommend using proper to assess probabilistic forecasts, a recommendation with which I wholeheartedly agree.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.