4
$\begingroup$

I have a question regarding calibration plot for a binary logistic regression model (calibrate) in the rms(regression modelling strategies) package. The Bias-corrected curve (see below) shows if the apparent fit of the model is overfited.

But how is the bias corrected curve obtained?

the explanation I found on page 270-271:

"The nonparametric estimate is evaluated at a sequence of predicted probability levels. Then the distances from the 45◦ line are compared with the differences when the current model is evaluated back on the whole sample (or omitted sample for cross-validation). The differences in the differences are estimates of overoptimism. After averaging over many replications, the predicted-value-specific differences are then subtracted from the apparent differences and an adjusted calibration curve is obtained.

My understanding of this is as follows:

1, fit a binary logistic model $M$ on the whole sample

2, fit a model $M_b$ from a bootstrap sample

3, predict probabilites from $M_B$ from the bootstrap sample and measure the distance to 45 degree line, $dist_B$, from the smoothed estimate(s) (evaluated at the predicted probabilies).

4, predict probabilities from $M_b$ on the whole sample and measure the distance to 45degree line,$dist_F$ from the smoothed estimate(s) (evaluated at the predited probabilites).

5, substract: $dist_B - dist_F$

6, complete steps 2-5 many times and average

7, substract whatever you get from (6) from the smoothed estimates you get from the predicted probabilites from $M$

Calibration curve

efrons method

  1. Construct a model in the original sample; determine the apparent performance on the data from the sample used to construct the model;
  2. Draw a bootstrap sample (Sample*) with replacement from the original sample

  3. Construct a model (Model*) in Sample*, replaying every step that was done in the original Sample, especially model specification steps such as selection of predictors from a larger set of candidate predictors. Determine the bootstrap performance as the apparent performance of Model* on Sample*;

  4. Apply Model* to the original Sample without any modification to determine the test performance;
  5. Calculate the optimism as the difference between bootstrap performance and test performance;
  6. Repeat steps 1–4 many times, at least 100, to obtain a stable estimate of the optimism;
  7. Subtract the optimism estimate (step 5) from the apparent performance (step 1) to obtain the optimism-corrected performance estimate.
$\endgroup$
4
$\begingroup$

The Regression Modeling Strategies book and course notes go into detail. This is the Efron-Gong optimism bootstrap in its original version. The bootstrap assumes that you are not using the outcome variable in any way to select the predictors in the model, or that you have done so using only backwards stepdown variable selection and you repeat this select for each bootstrap sample using calibrate(..., bw=TRUE).

Briefly, the bias that the bootstrap is estimating is the bias in how far the calibration curve at a single $x$-coordinate is from the line of identify.

Use at least 200 bootstrap repetitions. Were your sample size smaller, use 400.

$\endgroup$
7
  • $\begingroup$ I am not sure what the efron gongs method is; is it as the steps described above which I added afterwards? (clinical prediction models). I have the book but i cant find a detailed step by step describtion of how it is obtained that is the bias corrected curve @Frank Harell $\endgroup$ – J.doe Dec 19 '18 at 13:19
  • $\begingroup$ what is the line of identify? $\endgroup$ – J.doe Dec 19 '18 at 13:32
  • $\begingroup$ It's all in the book and course notes in detail. Look in the chapter dedicated to describing and validating the model, then look at the chapters on logistic regression and Cox regression. $\endgroup$ – Frank Harrell Dec 19 '18 at 17:14
  • $\begingroup$ What you listed as Efron's method in the edited original post is a duplication of what is listed initially. $\endgroup$ – Frank Harrell Dec 20 '18 at 12:47
  • $\begingroup$ Where is it listed initially? $\endgroup$ – J.doe Dec 20 '18 at 12:53

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.