Meaning of model calibration

Question

A previous post has discussed model discrimination very nicely. The post also briefly discussed calibration:

"When evaluating a risk model, calibration is also very important. To examine this, you will look at all patients with a risk score of around, e.g., 0.7, and see if approximately 70% of these actually were ill. Do this for each possible risk score (possibly using some sort of smoothing / local regression). Plot the results, and you’ll get a graphical measure of calibration.

If have a model with both good calibration and good discrimination, then you start to have good model"

I a bit confused about the meaning of calibration (see my previous post here) and would be grateful if someone could explain it in a similar fashion.

Answer 1

Let's suppose you have a set of training data and you have created a model that predicts the probability that a team will win a game. You did this e.g. by training a binary (win/loss) target on a set of input parameters. The model outputs a prediction, which is just the probability that the team will win the game.

You then generate such predictions using a separate test data set (on which you have not built your model).

You could then create "bins" or buckets of your predicted probabilities, say from 0 to 0.1, 0.1 to 0.2, ..., 0.9 to 1.0 and for all data rows that fall in to each bucket work out the actual mean "target" result (treating a win=1 and a loss=0).

If your model is "well-calibrated", the mean result in the bucket running between a predicted probability of 0 and 0.1, should be around 0.05 i.e. 5 wins if there were 100 rows of data with predicted probabilities between 0 and 0.1.

Your choice of bin size is dependent on how much data you have, but you would want there to be enough points in each bin such that the standard error on the mean of each bin is small.

Answer 2

If the model is well-calibrated the points will appear along the main diagonal on the diagnostic reliability diagrams(or calibration curves). The closer the more reliable the model.

If the points are below the diagonal, that indicates that the model has over-forecast; the probabilities are too large. And if the points are bbove the diagonal we can get to know that the model has under-forecast; the probabilities are too small.

We can see from the above line plot that the blue one, which represents logistic regression model, is more close to the diagonal than the orange line, which represent the random forest model, then we can say that the former model is more calibrated/reliable than the latter one.

And we can also see that for the logistic regression model whent he predictions are larger than 0.4 the probabiities are kind too small, indicating that the model has under-forecast.

References:

1) How and When to Use a Calibrated Classification Model with scikit-learn

2) A Guide to Calibration Plots in Python