How can I determine accuracy of past probability calculations? I do not study statistics but engineering, but this is a statistics question, and I hope you can lead me to what I need to learn to solve this problem.  
I have this situation where I calculate probabilities of 1000's of things happening in like 30 days.  If in 30 days I see what actually happened, how can I test to see how accurately I predicted?  These calculations result in probabilities and in actual values (ft).  What is the method for doing this?
Thanks,
CP
 A: In their classic book on the Federalist papers, Mosteller and Wallace argue for a log penalty function: you penalize yourself $-\log(p)$ when you predict an event with probability $p$ and it occurs; the penalty for it not occurring equals $-\log(1-p)$.  Thus, the penalty is high when whatever happens is unexpected according to your prediction.
Their argument in favor of this function rests on a simple natural criterion: "the penalty function should encourage the prediction of the correct probabilities if they are known."  Assuming the total penalty is summed over all predictions and there will be three or more of them, M&W claim that the log penalty function is the only one (up to affine transformation) for which the "expected penalty is minimized over all predictions" by the correct probabilities.
Following this, then, a good test for you to use is to track your accumulated log penalties.  If, after a long time (or by means of some independent oracle), you obtain accurate estimates of what the probabilities actually were, you can compare your penalty with the minimum possible one.  The average of that difference measures your long-run predictive performance (the lower the better).  This is an excellent way to compare two or more competing predictors, too.
A: What you're looking for are called Scoring Rules, which are ways of evaluating probabilistic forecasts. They were invented in the 1950s by weather forecasters, and there has been a been a bit of work on them in the statistics community, but I don't know of any books on the topic.
One thing you could do would be to bin the forecasts by probability range (e.g.: 0-5%, 5%-10%, etc.) and look at how many predicted events in that range occurred (if there are 40 events in the 0-5% range, and 20 occur, then your might have problems). If the events are independent, then you could compare these numbers to a binomial distribution.
