Lets use basketball as our example.
The use case:
- There is a model that predicts the probability that the favored team will win. The probability range then is necessarily constrained between 0.5 and 1 (obviously a team with less than 50% probability of winning isn't the favorite).
- Let say I have a database of say 3000 historical games where this model predicted the win probability for the favorite, and the actual result of that game. Naturally, 1 being a correct prediction (the favorite won), 0 being an incorrect prediction (the favorite lost).
- Likewise I suspect that this model is poor at predicting close games. Say anything between 0.5 and 0.55 win probability.
Can I run a single feature logistic regression (the feature being the predicted prob from the source model) against this dataset and use the resulting logistic regression curve to determine if the model is in fact poor at predicting close games?
It seems fair to me, to say that when a classically straightforward logistic regression based on the model's history predicts a probability lower than the one predicted by the source model, then our source model has likely been historically over-confident.