Determine accuracy of model which estimates probability of event

I'm modelling an event with two outcomes, a and b. I have created a model which estimates the probability that either a or b will happen (i.e. the model will calculate that a will happen with 40% chance and b will happen with 60% chance).

I have a large record of outcomes of trials with the estimates from the model. I would like to quantify how accurate the model is using this data - is this possible, and if so how?

• I might be wrong but I think you're interested in the training- and/or test-error of your model. See, for example: cs.ucla.edu/~falaki/pub/classification.pdf – Stijn Jan 3 '12 at 12:52
• @Stijn He's predicting the probability though rather than directly classifying as a or b, so I don't think those metrics are what he's asking for. – Michael McGowan Jan 3 '12 at 14:57
• Are you more interested in how well the model will eventually perform for classification (in which case ROC and AUC type of analysis seems most relevant (en.wikipedia.org/wiki/Receiver_operating_characteristic)? Or are you more interested in understanding how "calibrated" the probability predictions are (i.e. does P(Outcome = A)=60% really mean 60%, or just that outcome = A is more likely than the other outcomes... – DavidR Jan 3 '12 at 18:03
• It sounds like you want to know about probability scoring. – whuber Jan 3 '12 at 18:24
• Elvis, an article in the current issue of Decision Analysis drew my attention to probability scoring. It appears to build on substantial literature on the topic. (I don't have access to any more than the abstract, though, so I cannot comment on the article itself.) A cover paper by the journal's editors (which is freely available) mentions a number of previous papers on the same topic. – whuber Jan 3 '12 at 20:20

A common example of a scoring rule is the Brier score: $$BS = \frac{1}{N}\sum\limits _{t=1}^{N}(f_t-o_t)^2$$ where $f_t$ is the forecasted probability of the event happening and $o_t$ is 1 if the event did happen and 0 if it did not.