4
$\begingroup$

Calibration is important performance metric in predictive modeling. My question is about calibration plot in logistic regression. Observed values, say, in linear model are those which are actually observed for each unit. But how are "observed probabilities" in calibration plot obtained in logistic model where actual observed outcomes are binary? They are based on bootstrapping or cross-validation procedure as in calibrate in rms package when plotting calibration curve. But what is the actual procedure how these observed probabilities (y-axis below) are obtained?

enter image description here

Image from: How to interpret the basics of a logistic regression calibration plot please?

$\endgroup$
1
  • 1
    $\begingroup$ Probabilities cannot be observed: they can only be estimated relative to a model. See stats.stackexchange.com/questions/1525 for some discussion of this. $\endgroup$
    – whuber
    Commented Nov 23, 2020 at 16:21

1 Answer 1

3
$\begingroup$

The plot shows results of bootstrap validation, and this is really explained in the post you link too. The calculation is done with Frank Harrell's rms R package, and you can read its help file by typing ?rms::calibrate in an R session (assuming that package is installed). It is discussed in detail in the book Regression Modeling Strategies in section 10.9.

But the idea is simple enough, take the estimated linear predictor $X\hat{\beta}$ from a logistic regression, and estimate a new LR with that as predictor, that is, $$ P_c(Y=1 \mid X\hat{\beta})=\frac1{1+e^{-(\gamma_0+\gamma_1 X\hat{\beta})}} $$ If this is estimated on the same data, we will get $\gamma_0=0,\gamma_1=1$. But use bootstrapping, do the reestimation on each resample, and use the out-of-bag data points for validation. That is, calculate the predicted probabilities on the data points not in the resample. That is the basis of the calibration plot.

As explained in How to interpret the basics of a logistic regression calibration plot please? and Why does logistic regression produce well-calibrated models?, LR generally produces well-calibrated probabilities in the large, globally. That is because $$ \sum_i y_i = \sum_i \hat{p_i} $$ (assuming the model contains an intercept). But even then, there might be problems, for instance extreme probabilities could be to low (when large) and to low (when high). That would result in a calibration plot not following the ideal line at the diagonal $y=x$.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.