Is there a way to test a probabilistic prediction for an uncontrollable experiment?

Question

This is something that I've wondered about mostly in connection with weather forecasting, but I would like to ask the general question because it comes up from time to time in other contexts.

Suppose I have a process, denoted $F$, which, given some initial state $x$, produces an outcome $y$ from among a discrete set of possible outcomes: $F(x) \in\{y_1,\ldots,y_k\}$. The process is nondeterministic, so that running it multiple times with the same initial state $x$ will not always produce the same result. Also, suppose I have a theoretical model for the process which, given $x$, predicts a probability distribution over the various outcomes $y_1, \ldots, y_k$. I would like to test the hypothesis that the model is "correct." Or more precisely, given the following hypothesis:

For all initial states $x$, the distribution of results produced by the process converges to the probability distribution predicted by the model as the number of runs starting from state $x$ approaches infinity.

I would like to know at what significance level I can reject this hypothesis given a set of data points, each consisting of an initial state $X_i$ and the corresponding outcome $Y_i$.

If all the initial states $X_i$ are the same, then apparently I can do this by using a Pearson's $\chi^2$ test or Fisher's exact test. (Someone please correct me if this is wrong) If there are several different states among the $\{X_i\}$, then I can group the data by initial state and run Pearson's or Fisher's test on the group where $X_i=x_1$, and on the group where $X_i=x_2$, etc. all separately. In this way I could determine a significance level for the related hypothesis

For one specific initial state $x_j$, the distribution of results... (as above)

separately for each $x_j$. But I'm not interested in a bunch of individual significance levels for different initial states. I would like to have one statistical test that combines all my data and gives me one significance level for the model as a whole. This is essential because, in the extreme case, all the $X_i$ are distinct, so if I group my data by initial state, each group has only one element - which makes certainly Pearson's test, and to a large extent Fisher's as well, basically useless.

So, to sum up: is there a statistical test that will give me a significance level at which I can reject the hypothesis listed at the top of the question? If not in general, are there assumptions that make it possible? (e.g. does it matter if $x_j\in\mathbb{R}^n\forall x_j$ for some known, fixed, finite $n$?) Or is it known that this is impossible, and in that case, what's the reason?

In case it wasn't obvious, here's the canonical example I had in mind: $x$ would be the current state of the atmosphere (temperature, pressure, humidity, etc.) collected at weather stations around the US, plus satellite data, plus other stuff; the prediction would be something like P(rain)=0.6, P(cloudy)=0.3, P(sunny)=0.1 at some specified place and future time.

There are a few previously asked questions I've found that have relevant answers, especially this one and this one. Between them, there are several methods listed for comparing probabilistic predictions:

• Brier score
• Scoring rules
• Logistic regression
• ROC curve

But as far as I can tell (and I may well be wrong), all these provide comparative measures, not an actual statistical significance - in other words, the result of a given test (like the Brier score) is only useful for comparing against results of the same test on a different theoretical model.

---

I'm not particularly experienced with statistics, so if there are advanced concepts involved in answering this, it'd be great to have some references for further reading.

Answer 1

Standard statistical tests that produce p-values can rule out possibilities, but can never prove them true. So a test could use the fact that your model is correct as the null hypothesis and try to disprove it, but failure to disprove it does not prove it true. Even with all the x's the same Fisher's exact test and Pearson's $\chi^2$ test would not prove your probabilities to be correct, they could prove them wrong (up to a given type I error) or fail to disprove (but that does not prove it true).

Mulinomial regression creates a model to predict the probability of a discrete set of events, this looks most similar to what you specify, but it can never be proven to be exactly correct. You can use it to test if certain terms are statistically significant at improving the prediction, or if a complex model improves significantly over a simpler model.