A question which bothered me for some time, which I don't know how to address:
Every day, my weatherman gives a percentage chance of rain (let's assume its calculated to 9000 digits and he has never repeated a number). Every subsequent day, it either rains or does not rain.
I have years of data - pct chance vs rain or not. Given this weatherman's history, if he says tonight that tomorrow's chance of rain is X, then what's my best guess as to what the chance of rain really is?
In effect you are thinking of a model in which the true chance of rain, p, is a function of the predicted chance q: p = p(q). Each time a prediction is made, you observe one realization of a Bernoulli variate having probability p(q) of success. This is a classic logistic regression setup if you are willing to model the true chance as a linear combination of basis functions f1, f2, ..., fk; that is, the model says
Logit(p) = b0 + b1 f1(q) + b2 f2(q) + ... + bk fk(q) + e
with iid errors e. If you're agnostic about the form of the relationship (although if the weatherman is any good p(q) - q should be reasonably small), consider using a set of splines for the basis. The output, as usual, consists of estimates of the coefficients and an estimate of the variance of e. Given any future prediction q, just plug the value into the model with the estimated coefficients to obtain an answer to your question (and use the variance of e to construct a prediction interval around that answer if you like).
This framework is flexible enough to include other factors, such as the possibility of changes in the quality of predictions over time. It also lets you test hypotheses, such as whether p = q (which is what the weatherman implicitly claims).
Comparison of probability forecast for binary event (or discrete Random Variable) can be done upon the Brier score
but you can also use ROC curve since any probability forecast of this type can be transformed into a dicrimination procedure with a varying threshold Indeed you can say "it will rain" if your probability is greater than $\tau$ and evaluate the missed, false discovery,true discovery and true negatives for different values of $\tau$.
You should take a look at how the European center for medium range weather forecast (ECMWF does) .
When the forecast says "X percent chance of rain in (area)", it means that the numerical weather model has indicated rain in X percent of the area, for the time interval in question. For example, it would normally be accurate to predict "100 percent chance of rain in North America". Bear in mind that the models are good at predicting dynamics and poor at predicting thermodynamics.
The Brier Score approach is very simple and the most directly applicable way verify accuracy of a predicted outcome versus binary event.
Don't rely on just formulas ...plot the scores for different periods of time, data, errors, [weighted] rolling average of data, errors ... it's tough to say what visual analysis might reveal ... after you think you see something, you will better know what kind of hypothesis test to perform until AFTER you look at the data.
The Brier Score inherently assumes stability of the variation/underlying distributions weather and technology driving the forecasting models, lack of linearity, no bias, lack of change in bias ... it assumes that same general level of accuracy/inaccuracy is consistent. As climate changes in ways that are not yet understood, the accuracy of weather predictions would decrease; conversely, the scientists feeding information to the weatherman have more resources, more complete models, more computing power so perhaps the accuracy of the predictions would increase. Looking at the errors would tell something about stability, linearity and bias of the forecasts ... you may not have enough data to see trends; you may learn that stability, linearity and bias are not an issue. You may learn that weather forecasts are getting more accurate ... or not.
How about just binning the given predictions and taking the observed fractions as your estimate for each bin?
You can generalise this to a continuous model by weighing all the observations around your value of interest (say the prediction by tomorrow) by a Gaussian and seeing what the weighted average is.
You can guess a width to get you a given fraction of your data (or, say, never less than 100 points for a good estimate). Alternatively use a method such as cross-validation of max-likelihood to get the Gaussian width.
Do you want to know if his forecast is more accurate than another forecast? If so, you can look at basic accuracy metrics for probabilistic classification like cross-entropy, precision/recall, ROC curves, and the f1-score.
Determining if the forecast is objectively good is a different matter. One option is to look at calibration. Of all the days where he said that there would be a 90% chance of rain, did roughly 90% of those days have rain? Take all of the days where he has a forecast and then bucket them by his estimate of the probability of rain. For each bucket, calculate the percentage of the days where rain actually occurred. Then for each bucket plot the actual probability of rain against his estimate for the probability of rain. The plot will look like a straight line if the forecast is well calibrated.
