Determine accuracy of model which estimates probability of event

I'm modelling an event with two outcomes, a and b. I have created a model which estimates the probability that either a or b will happen (i.e. the model will calculate that a will happen with 40% chance and b will happen with 60% chance).

I have a large record of outcomes of trials with the estimates from the model. I would like to quantify how accurate the model is using this data - is this possible, and if so how?

• I might be wrong but I think you're interested in the training- and/or test-error of your model. See, for example: cs.ucla.edu/~falaki/pub/classification.pdf – Stijn Jan 3 '12 at 12:52
• @Stijn He's predicting the probability though rather than directly classifying as a or b, so I don't think those metrics are what he's asking for. – Michael McGowan Jan 3 '12 at 14:57
• Are you more interested in how well the model will eventually perform for classification (in which case ROC and AUC type of analysis seems most relevant (en.wikipedia.org/wiki/Receiver_operating_characteristic)? Or are you more interested in understanding how "calibrated" the probability predictions are (i.e. does P(Outcome = A)=60% really mean 60%, or just that outcome = A is more likely than the other outcomes... – DavidR Jan 3 '12 at 18:03
• It sounds like you want to know about probability scoring. – whuber Jan 3 '12 at 18:24
• Elvis, an article in the current issue of Decision Analysis drew my attention to probability scoring. It appears to build on substantial literature on the topic. (I don't have access to any more than the abstract, though, so I cannot comment on the article itself.) A cover paper by the journal's editors (which is freely available) mentions a number of previous papers on the same topic. – whuber Jan 3 '12 at 20:20

Suppose your model does indeed predict A has a 40% chance and B has a 60% chance. In some circumstances you might wish to convert this into a classification that B will happen (since it is more likely than A). Once converted into a classification, every prediction is either right or wrong, and there are a number of interesting ways to tally those right and wrong answers. One is straight accuracy (the percentage of right answers). Others include precision and recall or F-measure. As others have mentioned, you may wish to look at the ROC curve. Furthermore, your context may supply a specific cost matrix that rewards true positives differently from true negatives and/or penalizes false positives differently from false negatives.

However, I don't think that's what you are really looking for. If you said B has a 60% chance of happening and I said it had a 99% chance of happening, we have very different predictions even though they would both get mapped to B in a simple classification system. If A happens instead, you are just kind of wrong while I am very wrong, so I'd hope that I would receive a stiffer penalty than you. When your model actually produces probabilities, a scoring rule is a measure of performance of your probability predictions. Specifically you probably want a proper scoring rule, meaning that the score is optimized for well-calibrated results.

A common example of a scoring rule is the Brier score: $$BS = \frac{1}{N}\sum\limits _{t=1}^{N}(f_t-o_t)^2$$ where $f_t$ is the forecasted probability of the event happening and $o_t$ is 1 if the event did happen and 0 if it did not.

Of course the type of scoring rule you choose might depend on what type of event you are trying to predict. However, this should give you some ideas to research further.

I'll add a caveat that regardless of what you do, when assessing your model this way I suggest you look at your metric on out-of-sample data (that is, data not used to build your model). This can be done through cross-validation. Perhaps more simply you can build your model on one dataset and then assess it on another (being careful not to let inferences from the out-of-sample spill into the in-sample modeling).