Can I use a logistic regression as a calibration curve?

Question

Lets use basketball as our example.

The use case:

• There is a model that predicts the probability that the favored team will win. The probability range then is necessarily constrained between 0.5 and 1 (obviously a team with less than 50% probability of winning isn't the favorite).
• Let say I have a database of say 3000 historical games where this model predicted the win probability for the favorite, and the actual result of that game. Naturally, 1 being a correct prediction (the favorite won), 0 being an incorrect prediction (the favorite lost).
• Likewise I suspect that this model is poor at predicting close games. Say anything between 0.5 and 0.55 win probability.

Can I run a single feature logistic regression (the feature being the predicted prob from the source model) against this dataset and use the resulting logistic regression curve to determine if the model is in fact poor at predicting close games?

It seems fair to me, to say that when a classically straightforward logistic regression based on the model's history predicts a probability lower than the one predicted by the source model, then our source model has likely been historically over-confident.

Answer 1

If you were going to do this, it would make a lot of sense to have the feature be the log-odds rather than the predicted probability. Your null hypothesis should be that the model is perfect, and so some values of the parameters in your regression model need to correspond to that case. Since in the regression model $y \sim x$, $y$ is interpreted as log-odds of the target variable, then $x$ should be the predicted log-odds so that a perfectly calibrated model corresponds to $y = x$ -- any significant discrepancy of the fit slope and intercept from $(0,0)$ would be evidence of miscalibration. It's easy to see that if the feature $x$ is the probability, then the model can't exactly fit the perfectly calibrated model, because the log odds is a linear function over the domain $[0,1]$ and is therefore bounded itself, hence having a minimum and maximum predicted probability when the underlying model could make 0% and 100% predictions.