How to evaluate the accuracy of probability of a large set of non-repeatable events? Assuming there are a large set ($N>1000$) of independent events $E_i$ ($i=1,2,\dots,N$), each having $M$ different outcomes. For each event $E_i$, I have an estimated discrete probability $\hat P_j(E_i)$ of outcome $j$, ($j=1,2,\dots,M)$, which is not necessarily the same as the 'true' probability $P_j(E_i)$. The events are non-repeatable and only happen once, and have one real outcome $O_i \in \{1,2,\cdots,M\}$. How can we evaluate the accuracy of the estimated probabilities? If someone claims that the set of estimation $\{\hat P_j(E_i)|1 \leq i \leq N, 1 \leq j \leq M \}$ corresponds to the "true" probabilities $\{P_j(E_i)|1 \leq i \leq N, 1 \leq j \leq M \}$, given infinite number of independent events $N \rightarrow +\infty$, how can I verify whether the claim is true, if an infinite computational power is available?
In a more concrete context, we can imagine that the events are sports events with three possible outcomes (home win, away win and draw) and the estimated probabilities are the predictions of sports websites, which are also reflected in the odds given by the sports bookmakers. How can I evaluate how reliable the predictions (given in terms of probabilities) of a certain website are in long run?
 A: Proving that you have the exact correct probability values will be impossible. However, probability validation does exist and seems to be what you seek.
Probability validation aims to assess how accurate claimed probability values are. For instance, if I keep claiming that there is a $50\%$ chance of an event happening, and it almost never happens, it is fair to say that my claimed probability is wrong. If I claim $80\%$ for one event and $10\%$ for another, which happen $60\%$ and $40\%$ of the time, respectively, my claims are inaccurate. The answer
Probability validation as discussed in the link takes that idea to the extreme, where you have a bunch of events with different probabilities and only observe the discrete outcome. At the same time, you make your predictions/guesses about the probability with which the events will occur.
I found the linked answer by Demetri Pananos extremely helpful in understanding what happens under the hood of the R function discussed there, so much so that I was able to translate it to Python for a customer that wanted its functionality but had to operate in a Python environment; I even extended the Python implementation to handle multiple possible outcomes like you seem to have in your work. Thus, I will leave to the link how exactly probability validation works and just say that this seems to be what you aim to do. (Indeed, that is just one thought about how to do it. There might be other reasonable approaches, perhaps even superior approaches.)
There could be hypothesis testing approaches to assessing probability. However, such an approach would suffer from the usual issues of hypothesis testing: just because you reject doesn’t mean that the deviation from the ideal is enough for us to care, and failure to reject does not mean that we have accurate probability predictions. Consequently, graphical approaches are quite useful.
Probability calibration, which is mentioned in the link, is a related but distinct idea.
