# Interpretation of calibration curve

I have a step-wise derived binary logistic regression model. I have used the calibrate(, bw=200, bw=TRUE) function in the rms package in R to estimate its future calibration. The output is given below and it shows the bootstrap overfitting-corrected calibration curve estimate for the backward step-down logistic model. However, I am not sure how to interpret it.

I understand that calibration refers to whether the future predicted probabilities agree with the observed probabilities. Prediction models suffer that predictions for new subjects are too extreme (i.e, that the observed probability of the outcome is higher than predicted for low-risk subjects and lower than predicted for high-risk subjects). This is seen by tracing the dotted curve which is higher than the ideal (dashed) for low-risk group, and lower than the ideal for high-risk group.

Using the same reasoning, the bias-corrected curve seems to be worse, in the sense that it produces even more extreme probabilities. Is my interpretation correct?

• bw=200 should read bw=TRUE Commented Sep 2, 2017 at 13:11

The curve labeled bias-corrected appears to be "over confident": its predictions for Predicted P(Class=1)<0.5 are too low and its predictions for Predicted P(Class=1)>0.5 are too high relative to Actual probability.
This is also the case for the curve labeled apparent, except at the extremes (roughly: x<=0.28 or x>=0.9) it appears to actually be $less$ confident.
I am unsure of the details of the bias correcting method in rms, but I don't think the result is necessarily "worse"; With the correction, the probability estimates are parallel to the ideal. In other words, although it's known that the model is slightly overconfident, we can say that the difference between its mean prediction for a population with P(class)=0.2 is half of its mean prediction for a population with P(class)=0.4, which was not the case before and probably what one would hope for.