1
$\begingroup$

Assuming there are a large set ($N>1000$) of independent events $E_i$ ($i=1,2,\dots,N$), each having $M$ different outcomes. For each event $E_i$, I have an estimated discrete probability $\hat P_j(E_i)$ of outcome $j$, ($j=1,2,\dots,M)$, which is not necessarily the same as the 'true' probability $P_j(E_i)$. The events are non-repeatable and only happen once, and have one real outcome $O_i \in \{1,2,\cdots,M\}$. How can we evaluate the accuracy of the estimated probabilities? If someone claims that the set of estimation $\{\hat P_j(E_i)|1 \leq i \leq N, 1 \leq j \leq M \}$ corresponds to the "true" probabilities $\{P_j(E_i)|1 \leq i \leq N, 1 \leq j \leq M \}$, given infinite number of independent events $N \rightarrow +\infty$, how can I verify whether the claim is true, if an infinite computational power is available?

In a more concrete context, we can imagine that the events are sports events with three possible outcomes (home win, away win and draw) and the estimated probabilities are the predictions of sports websites, which are also reflected in the odds given by the sports bookmakers. How can I evaluate how reliable the predictions (given in terms of probabilities) of a certain website are in long run?

Improve this question
$\endgroup$
3
  • $\begingroup$ Please clarify your specific problem or provide additional details to highlight exactly what you need. As it's currently written, it's hard to tell exactly what you're asking. $\endgroup$
    – utobi
    Commented Oct 30, 2022 at 21:09
  • $\begingroup$ I have added sports outcome predictions as an example, I am not sure whether that would be sufficient. $\endgroup$
    – Sato
    Commented Oct 30, 2022 at 21:20
  • $\begingroup$ For two different models, say $\hat{P}_{j\in 1...M}^{(1)}$ and $\hat{P}_{j\in 1...M}^{(2)}$, the Bayes factor would be useful for comparison. $\endgroup$
    – jblood94
    Commented Nov 3, 2022 at 19:56

1 Answer 1

Reset to default
0
$\begingroup$

Proving that you have the exact correct probability values will be impossible. However, probability validation does exist and seems to be what you seek.

Probability validation aims to assess how accurate claimed probability values are. For instance, if I keep claiming that there is a $50\%$ chance of an event happening, and it almost never happens, it is fair to say that my claimed probability is wrong. If I claim $80\%$ for one event and $10\%$ for another, which happen $60\%$ and $40\%$ of the time, respectively, my claims are inaccurate. The answer

Probability validation as discussed in the link takes that idea to the extreme, where you have a bunch of events with different probabilities and only observe the discrete outcome. At the same time, you make your predictions/guesses about the probability with which the events will occur.

I found the linked answer by Demetri Pananos extremely helpful in understanding what happens under the hood of the R function discussed there, so much so that I was able to translate it to Python for a customer that wanted its functionality but had to operate in a Python environment; I even extended the Python implementation to handle multiple possible outcomes like you seem to have in your work. Thus, I will leave to the link how exactly probability validation works and just say that this seems to be what you aim to do. (Indeed, that is just one thought about how to do it. There might be other reasonable approaches, perhaps even superior approaches.)

There could be hypothesis testing approaches to assessing probability. However, such an approach would suffer from the usual issues of hypothesis testing: just because you reject doesn’t mean that the deviation from the ideal is enough for us to care, and failure to reject does not mean that we have accurate probability predictions. Consequently, graphical approaches are quite useful.

Probability calibration, which is mentioned in the link, is a related but distinct idea.

Improve this answer
$\endgroup$
5
  • $\begingroup$ Thanks for your answer! Would you mind showing the Python implementation? For I cannot read R, but only Python. $\endgroup$
    – Sato
    Commented Oct 30, 2022 at 21:30
  • $\begingroup$ I don’t think I can post software that I wrote for someone. However, Demetri and I give the entire implementation in English (or at least in statistics). $\endgroup$
    – Dave
    Commented Oct 30, 2022 at 21:35
  • $\begingroup$ I see. I have actually found another question in Stack Exchange, do you think it represents a similar idea? math.stackexchange.com/questions/1133635/… $\endgroup$
    – Sato
    Commented Oct 30, 2022 at 22:23
  • $\begingroup$ And the initial idea I had was to verify the reliability of the set of probabilities using simulations, by somehow "averaging" a large amount of random guesses based on the estimated probability distribution (I don't have a complete idea yet), don't know whether such approaches would makes sense for this purpose. For example, in sports betting, betting based on "true odds" means having zero edges against bookmakers, so in average, the expected earnings should be zero in the long run, which should be an indication that the odds (which are reciprocals of probabilities) are actually true. $\endgroup$
    – Sato
    Commented Oct 30, 2022 at 22:27
  • $\begingroup$ The trouble with such an approach is that, if you overestimate the low probabilities and underestimate the high probabilities, you average out to having gotten it right. With a probability validation plot, you’d catch that with just a glance at the graph. $\endgroup$
    – Dave
    Commented Oct 30, 2022 at 22:31

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.