2
$\begingroup$

In my mind, (I think) I understand the difference between prediction and uncertainty about that prediction. The prediction comes from a model (say a LPM or a Probit) and the uncertainty is related to how wide the confidence intervals are around a particular prediction. However I am having a hard time explaining to a friend that prediction is different from uncertainty, in particular when the prediction is about a probability. For example, my friend is interested in sports and he posits that if you (ie a model) predicts that Team A will win over Team B with 95% chance, it should be very unlikely that Team B ends up winning. He is frustrated when Team B wins, and I tell him that (1) there still was a 5% chance of Team B winning, but also (2) that there could have been a lot of uncertainty around that 95% prediction of A winning. He replies that if there is a lot of uncertainty in the prediction, then it should not have been 95% in the first place. This is when our conversation stalls. I keep repeating that prediction is not the same as uncertainty, while he keeps digging on "then the probability of A winning should not have been so high." How to best explain the difference to my friend?

Also, probably we are falling into some fallacy, and if so I'd like to know which one.

This issue is also probably related to uncertainty of the model vs uncertainty of the prediction. Where can I find some reading material about that?

$\endgroup$
3
  • $\begingroup$ Any chance this has to do with the college basketball tournament going on now? $\endgroup$
    – Dave
    Mar 24, 2021 at 10:23
  • $\begingroup$ Not really. My friend is into baseball. $\endgroup$
    – amrods
    Mar 24, 2021 at 10:26
  • 1
    $\begingroup$ I think this is a difficult concept to understand. It reminds me of Donald Rumsfeld's quote "There are known knowns. There are things we know we know. We also know there are known unknowns. That is to say, we know there are some things we do not know. But there are also unknown unknowns, the ones we don't know we don't know." Sometimes you know the probability is 0.95 and sometimes you know that you don't know the probability is 0.95 but that is the best guess. $\endgroup$
    – John L
    Mar 24, 2021 at 15:23

1 Answer 1

2
$\begingroup$

Prediction usually refers to the outcome, in this example whether team A wins or loses a particular game against team B. The 95% chance that team A will win is referred to as the estimated probability.

Let's consider the simple example where there is no possibility of tie (e.g., sudden death). There are only two outcomes: A - Team A wins, and B - Team B wins which is the same as Team A does not win. We have the probabilities P(A), and P(B) = 1- P(A). P(A) is unknown to us, but from a model and some data (previous years of Team A playing against Team B, players, coaches, etc.) you have estimated P(A), call it $\hat{p}_A$. If $\hat{p}_A = 0.95$, then necessarily $\hat{p}_B = 0.05$, and here, I think, lies the key to get you going again. Let the uncertainty of $\hat{p}_A = 0.95$ suggest a confidence interval for $\hat{p}_A$ of $(0.60, 0.99)$ (these are made up numbers based on a hypothetical model), and let there be another Team C, also playing against Team B, with estimated probability of winning $\hat{p}_C = 0.95$ and confidence interval $(0.92,0.97)$. I would certainly put my money on Team C instead of Team A.

But why not just say $\hat{p}_A = 0.60$ and $\hat{p}_C = 0.92$, as your friend suggests? The corresponding probability of Team B winning over Team A will then necessarily be $\hat{p}_B = 0.40$, and will raise the question "Why is $\hat{p}_B$ so high when they literally never win?". Say you are following Team A through the season, with multiple games against Team B. At the end of the season, you expect the estimated probability to reflect the proportion of games that Team A won, and then the original $\hat{p}_A = 0.95$ is a better guess than 0.60, but still not a very good one.

$\endgroup$
1
  • $\begingroup$ I think this is getting to it. It exactly addresses the example I proposed. But what if the events are one-shot, say elections? How can that be addressed? $\endgroup$
    – amrods
    Apr 15, 2021 at 10:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.