12
$\begingroup$

Question: I have fitted a probabilistic model (bayesian network) for modeling a binary outcome variable. I would like to create a high-resolution calibration plot (e.g. spline) corrected for overfitting with bootstrapping. Is there a standard procedure for calculating such a curve?

Considerations: I could do this easily with train/test splitting, but I would rather not throw away any data as I have less than 20,000 samples. So I naturally thought about bootstrapping. I know that one such function (calibrate) is implemented in Frank Harrell's rms package, but unfortunately the model I use is not supported by the package.

Bonus question: is it possible to recalibrate a miscalibrated model with bootstrapping? The reason I ask this is that I tried to recalibrate a model by

  1. split data in train/test
  2. fitting model to train set
  3. recalibrate model to train set (with a cubic spline)
  4. evaluate calibration on test set

The models recalibrated in the fashion above were perfectly calibrated on the train set but not so much on the test set, which probably indicates mild overfitting. I also tried further splitting the test set, calibrating on one split and evaluating the calibration on the second split. I got better results (still not perfectly calibrated though), but the sets became quite small (~1000 samples) and thus the calibration unreliable

$\endgroup$
9
  • 1
    $\begingroup$ Not sure what's available in R, but Python's sklearn has an excellent probability calibration module which supports cross-validation and isotonic/monotonic regression, which are key for high quality probability calibration.Might be a good place to get some ideas. scikit-learn.org/stable/modules/calibration.html $\endgroup$
    – olooney
    Commented May 17, 2018 at 16:23
  • $\begingroup$ Thank you for your answer! I am bound to R and not particularly keen on starting with reticulate. I could implement the procedure myself if I knew it, but I haven't found it anywhere.. I guess I am just hoping that prof Harrell will see this question :D $\endgroup$ Commented May 17, 2018 at 20:42
  • 1
    $\begingroup$ Please include a (small) data example to illustrate what you're talking about. Thank you. $\endgroup$
    – Jim
    Commented May 24, 2018 at 21:57
  • $\begingroup$ You should consider taking a look at this introduction to the rms package (and its calibrate function) in R: r-bloggers.com/introduction-to-the-rms-package $\endgroup$
    – rpatel
    Commented May 25, 2018 at 15:57
  • $\begingroup$ Hi rpatel, thanks for the suggestion. I had mentioned the rms::calibrate function in my original question, noting that it does not support the model class I am using. I also own Harrell's Regression Modelling Strategies book, but I am unable to find any detailed description on how the calibrate function works. $\endgroup$ Commented May 28, 2018 at 7:58

1 Answer 1

10
$\begingroup$

After discussing with prof Frank Harrell by email, I devised the following procedure for estimating the optimism-corrected calibration curve, partially based on his Tutorial in Biostatistics (STATISTICS IN MEDICINE, VOL. 15,361-387 (1996)):

  1. fit a risk prediction model on all data
  2. fit a flexible model (gam with spline and logit link) to the model's predicted probabilities vs outcome, and query the gam at a grid of predicted probabilities $p=(0.01,0.02,...,0.99)$. This is the apparent calibration curve and we call it $cal_{app}$
  3. draw bootstrap sample with replacement, same size of original data
  4. fit risk prediction model on bootstrap sample
  5. use the bootstrap model to predict probabilities from the bootstrap sample, fit a gam between the predicted probabilities and the outcome, and query the gam at a grid of predicted probabilities (let us call these points $cal_{boot}$)
  6. use the bootstrap model to predict probabilities from the original sample, fit a gam between the predicted probabilities and the outcome, and query the gam at a grid of predicted probabilities obtaining a calibration curve ($cal_{orig}$)
  7. compute the optimism at every point $p$ of the grid like so $$Optimism(p)=cal_{boot}(p) - cal_{orig}(p)$$
  8. repeat steps 3-7 some 100 times, average the optimism at each point $p$
  9. compute the optimism corrected calibration like so $$cal_{corr}(p)=cal_{app}(p)-<Optimism(p)>$$

Important note: The procedure above is inspired by Harrell's work and my discussion with him, but all errors are mine alone.

$\endgroup$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service and acknowledge you have read our privacy policy.

Not the answer you're looking for? Browse other questions tagged or ask your own question.