This is something that I've wondered about mostly in connection with weather forecasting, but I would like to ask the general question because it comes up from time to time in other contexts.
Suppose I have a process, denoted $F$, which, given some initial state $x$, produces an outcome $y$ from among a discrete set of possible outcomes: $F(x) \in\{y_1,\ldots,y_k\}$. The process is nondeterministic, so that running it multiple times with the same initial state $x$ will not always produce the same result. Also, suppose I have a theoretical model for the process which, given $x$, predicts a probability distribution over the various outcomes $y_1, \ldots, y_k$. I would like to test the hypothesis that the model is "correct." Or more precisely, given the following hypothesis:
For all initial states $x$, the distribution of results produced by the process converges to the probability distribution predicted by the model as the number of runs starting from state $x$ approaches infinity.
I would like to know at what significance level I can reject this hypothesis given a set of data points, each consisting of an initial state $X_i$ and the corresponding outcome $Y_i$.
If all the initial states $X_i$ are the same, then apparently I can do this by using a Pearson's $\chi^2$ test or Fisher's exact test. (Someone please correct me if this is wrong) If there are several different states among the $\{X_i\}$, then I can group the data by initial state and run Pearson's or Fisher's test on the group where $X_i=x_1$, and on the group where $X_i=x_2$, etc. all separately. In this way I could determine a significance level for the related hypothesis
For one specific initial state $x_j$, the distribution of results... (as above)
separately for each $x_j$. But I'm not interested in a bunch of individual significance levels for different initial states. I would like to have one statistical test that combines all my data and gives me one significance level for the model as a whole. This is essential because, in the extreme case, all the $X_i$ are distinct, so if I group my data by initial state, each group has only one element - which makes certainly Pearson's test, and to a large extent Fisher's as well, basically useless.
So, to sum up: is there a statistical test that will give me a significance level at which I can reject the hypothesis listed at the top of the question? If not in general, are there assumptions that make it possible? (e.g. does it matter if $x_j\in\mathbb{R}^n\forall x_j$ for some known, fixed, finite $n$?) Or is it known that this is impossible, and in that case, what's the reason?
In case it wasn't obvious, here's the canonical example I had in mind: $x$ would be the current state of the atmosphere (temperature, pressure, humidity, etc.) collected at weather stations around the US, plus satellite data, plus other stuff; the prediction would be something like P(rain)=0.6, P(cloudy)=0.3, P(sunny)=0.1 at some specified place and future time.
There are a few previously asked questions I've found that have relevant answers, especially this one and this one. Between them, there are several methods listed for comparing probabilistic predictions:
But as far as I can tell (and I may well be wrong), all these provide comparative measures, not an actual statistical significance - in other words, the result of a given test (like the Brier score) is only useful for comparing against results of the same test on a different theoretical model.
I'm not particularly experienced with statistics, so if there are advanced concepts involved in answering this, it'd be great to have some references for further reading.