Compare two sets of probabilities to outcome data

Question

Suppose there are two predictive models that both output the probability that the home team wins a given match. Then suppose there is data for thousands of matches, in the format:

MODEL A, MODEL B, RESULT 0.3, 0.4, W 0.4, 0.5, L 0.2, 0.3, W ...

Also suppose nothing is known about the models from which these values arise.

1. What is the simplest way of comparing the accuracy of the two models?
2. Is there an "industry standard" metric that should be used under these circumstances?

Answer 1

There are many methods available, especially if you start by examining the individual accuracy of each of the two methods. See for example my val.prob function in the R rms package (documentation here). This will draw a smooth nonparametric calibration curve to assess the absolute forecast accuracy and will give you the powerful Spiegelhalter test for testing for calibration accuracy.

For an over-simplified calibration model you can use binary logistic regression to play the logit of one of the model predicted probabilities against observed 0/1 outcomes and test the slopes against 1.0 and the intercept against 0.0. If the slope is near 1.0 but the intercept is much different from 0.0 that would be a "miscalibration in the large".

Things are clear if one of the methods is well calibrated and the other isn't. Otherwise you can easily use logistic regression to test whether one method adds predictive information to the other. Material related to that is at my blog article https://fharrell.com/post/addvalue. This can be as simple as fitting a binary logistic model with two predictors, which are the logit of the probability forecast for each of the two methods.

Answer 2

1. You can use confusion Matrix; refer wiki for more details in your case you can assume probability above 0.5 as win.

2. For absolute results you can directly compare percentage accuracy.