How to best explain prediction vs uncertainty in the case of probabilities

Question

In my mind, (I think) I understand the difference between prediction and uncertainty about that prediction. The prediction comes from a model (say a LPM or a Probit) and the uncertainty is related to how wide the confidence intervals are around a particular prediction. However I am having a hard time explaining to a friend that prediction is different from uncertainty, in particular when the prediction is about a probability. For example, my friend is interested in sports and he posits that if you (ie a model) predicts that Team A will win over Team B with 95% chance, it should be very unlikely that Team B ends up winning. He is frustrated when Team B wins, and I tell him that (1) there still was a 5% chance of Team B winning, but also (2) that there could have been a lot of uncertainty around that 95% prediction of A winning. He replies that if there is a lot of uncertainty in the prediction, then it should not have been 95% in the first place. This is when our conversation stalls. I keep repeating that prediction is not the same as uncertainty, while he keeps digging on "then the probability of A winning should not have been so high." How to best explain the difference to my friend?

Also, probably we are falling into some fallacy, and if so I'd like to know which one.

This issue is also probably related to uncertainty of the model vs uncertainty of the prediction. Where can I find some reading material about that?

Answer 1

Prediction usually refers to the outcome, in this example whether team A wins or loses a particular game against team B. The 95% chance that team A will win is referred to as the estimated probability.

Let's consider the simple example where there is no possibility of tie (e.g., sudden death). There are only two outcomes: A - Team A wins, and B - Team B wins which is the same as Team A does not win. We have the probabilities P(A), and P(B) = 1- P(A). P(A) is unknown to us, but from a model and some data (previous years of Team A playing against Team B, players, coaches, etc.) you have estimated P(A), call it $\hat{p}_A$. If $\hat{p}_A = 0.95$, then necessarily $\hat{p}_B = 0.05$, and here, I think, lies the key to get you going again. Let the uncertainty of $\hat{p}_A = 0.95$ suggest a confidence interval for $\hat{p}_A$ of $(0.60, 0.99)$ (these are made up numbers based on a hypothetical model), and let there be another Team C, also playing against Team B, with estimated probability of winning $\hat{p}_C = 0.95$ and confidence interval $(0.92,0.97)$. I would certainly put my money on Team C instead of Team A.

But why not just say $\hat{p}_A = 0.60$ and $\hat{p}_C = 0.92$, as your friend suggests? The corresponding probability of Team B winning over Team A will then necessarily be $\hat{p}_B = 0.40$, and will raise the question "Why is $\hat{p}_B$ so high when they literally never win?". Say you are following Team A through the season, with multiple games against Team B. At the end of the season, you expect the estimated probability to reflect the proportion of games that Team A won, and then the original $\hat{p}_A = 0.95$ is a better guess than 0.60, but still not a very good one.