# Is there a way to test a probabilistic prediction for an uncontrollable experiment?

This is something that I've wondered about mostly in connection with weather forecasting, but I would like to ask the general question because it comes up from time to time in other contexts.

Suppose I have a process, denoted $F$, which, given some initial state $x$, produces an outcome $y$ from among a discrete set of possible outcomes: $F(x) \in\{y_1,\ldots,y_k\}$. The process is nondeterministic, so that running it multiple times with the same initial state $x$ will not always produce the same result. Also, suppose I have a theoretical model for the process which, given $x$, predicts a probability distribution over the various outcomes $y_1, \ldots, y_k$. I would like to test the hypothesis that the model is "correct." Or more precisely, given the following hypothesis:

For all initial states $x$, the distribution of results produced by the process converges to the probability distribution predicted by the model as the number of runs starting from state $x$ approaches infinity.

I would like to know at what significance level I can reject this hypothesis given a set of data points, each consisting of an initial state $X_i$ and the corresponding outcome $Y_i$.

If all the initial states $X_i$ are the same, then apparently I can do this by using a Pearson's $\chi^2$ test or Fisher's exact test. (Someone please correct me if this is wrong) If there are several different states among the $\{X_i\}$, then I can group the data by initial state and run Pearson's or Fisher's test on the group where $X_i=x_1$, and on the group where $X_i=x_2$, etc. all separately. In this way I could determine a significance level for the related hypothesis

For one specific initial state $x_j$, the distribution of results... (as above)

separately for each $x_j$. But I'm not interested in a bunch of individual significance levels for different initial states. I would like to have one statistical test that combines all my data and gives me one significance level for the model as a whole. This is essential because, in the extreme case, all the $X_i$ are distinct, so if I group my data by initial state, each group has only one element - which makes certainly Pearson's test, and to a large extent Fisher's as well, basically useless.

So, to sum up: is there a statistical test that will give me a significance level at which I can reject the hypothesis listed at the top of the question? If not in general, are there assumptions that make it possible? (e.g. does it matter if $x_j\in\mathbb{R}^n\forall x_j$ for some known, fixed, finite $n$?) Or is it known that this is impossible, and in that case, what's the reason?

In case it wasn't obvious, here's the canonical example I had in mind: $x$ would be the current state of the atmosphere (temperature, pressure, humidity, etc.) collected at weather stations around the US, plus satellite data, plus other stuff; the prediction would be something like P(rain)=0.6, P(cloudy)=0.3, P(sunny)=0.1 at some specified place and future time.

There are a few previously asked questions I've found that have relevant answers, especially this one and this one. Between them, there are several methods listed for comparing probabilistic predictions:

But as far as I can tell (and I may well be wrong), all these provide comparative measures, not an actual statistical significance - in other words, the result of a given test (like the Brier score) is only useful for comparing against results of the same test on a different theoretical model.

I'm not particularly experienced with statistics, so if there are advanced concepts involved in answering this, it'd be great to have some references for further reading.

• I wasn't really sure what tags to put on this, so perhaps someone could help out with that. Feb 17, 2012 at 22:42
• I am wondering whether existing threads already answer this question. Search for "weather" and look especially at stats.stackexchange.com/questions/2275/… and stats.stackexchange.com/questions/1875/….
– whuber
Feb 17, 2012 at 22:53
• Thanks, I didn't see either of those while I was looking for duplicates (probably because I didn't know what exactly to look for). The answers to the latter question seem relevant. Of course, I tried to be more precise in formulating this question (and I'm asking something more general than just how to rate weather forecasters), but it wouldn't surprise me if anything that could be written as an answer to this has already been posted there. Feb 17, 2012 at 23:57
• Consider, then, editing this question to emphasize the way(s) in which it differs from the previous ones, so we can focus our replies on the additional information you need.
– whuber
Feb 18, 2012 at 1:24
• OK, I don't have time right now but I'll work on it soon. I'll have to review the answers to the other questions, though, and it may turn out that they do provide exactly what I'm looking for - in which case I'll just comment here that this can be closed as a duplicate (or deleted). Feb 18, 2012 at 1:44

Standard statistical tests that produce p-values can rule out possibilities, but can never prove them true. So a test could use the fact that your model is correct as the null hypothesis and try to disprove it, but failure to disprove it does not prove it true. Even with all the x's the same Fisher's exact test and Pearson's $\chi^2$ test would not prove your probabilities to be correct, they could prove them wrong (up to a given type I error) or fail to disprove (but that does not prove it true).