In effect you are thinking of a model in which the true chance of rain, p, is a function of the predicted chance q: p = p(q). Each time a prediction is made, you observe one realization of a Bernoulli variate having probability p(q) of success. This is a classic logistic regression setup if you are willing to model the true chance as a linear combination of basis functions f1, f2, ..., fk; that is, the model says
Logit(p) = b0 + b1 f1(q) + b2 f2(q) + ... + bk fk(q) + e
with iid errors e. If you're agnostic about the form of the relationship (although if the weatherman is any good p(q) - q should be reasonably small), consider using a set of splines for the basis. The output, as usual, consists of estimates of the coefficients and an estimate of the variance of e. Given any future prediction q, just plug the value into the model with the estimated coefficients to obtain an answer to your question (and use the variance of e to construct a prediction interval around that answer if you like).
This framework is flexible enough to include other factors, such as the possibility of changes in the quality of predictions over time. It also lets you test hypotheses, such as whether p = q (which is what the weatherman implicitly claims).