How to estimate a calibration curve with bootstrap (R) Question: I have fitted a probabilistic model (bayesian network) for modeling a binary outcome variable. I would like to create a high-resolution calibration plot (e.g. spline) corrected for overfitting with bootstrapping. Is there a standard procedure for calculating such a curve?
Considerations: I could do this easily with train/test splitting, but I would rather not throw away any data as I have less than 20,000 samples. So I naturally thought about bootstrapping. I know that one such function (calibrate) is implemented in Frank Harrell's rms package, but unfortunately the model I use is not supported by the package.
Bonus question: is it possible to recalibrate a miscalibrated model with bootstrapping? The reason I ask this is that I tried to recalibrate a model by


*

*split data in train/test

*fitting model to train set

*recalibrate model to train set (with a cubic spline)

*evaluate calibration on test set


The models recalibrated in the fashion above were perfectly calibrated on the train set but not so much on the test set, which probably indicates mild overfitting. I also tried further splitting the test set, calibrating on one split and evaluating the calibration on the second split. I got better results (still not perfectly calibrated though), but the sets became quite small (~1000 samples) and thus the calibration unreliable
 A: After discussing with prof Frank Harrell by email, I devised the following procedure for estimating the optimism-corrected calibration curve, partially based on his Tutorial in Biostatistics (STATISTICS IN MEDICINE, VOL. 15,361-387 (1996)):


*

*fit a risk prediction model on all data

*fit a flexible model (gam with spline and logit link) to the model's predicted probabilities vs outcome, and query the gam at a grid of predicted probabilities $p=(0.01,0.02,...,0.99)$. This is the apparent calibration curve and we call it $cal_{app}$

*draw bootstrap sample with replacement, same size of original data

*fit risk prediction model on bootstrap sample

*use the bootstrap model to predict probabilities from the bootstrap sample, fit a gam between the predicted probabilities and the outcome, and query the gam at a grid of predicted probabilities (let us call these points $cal_{boot}$)

*use the bootstrap model to predict probabilities from the original sample, fit a gam between the predicted probabilities and the outcome, and query the gam at a grid of predicted probabilities obtaining a calibration curve ($cal_{orig}$)

*compute the optimism at every point $p$ of the grid like so $$Optimism(p)=cal_{boot}(p) - cal_{orig}(p)$$

*repeat steps 3-7 some 100 times, average the optimism at each point $p$

*compute the optimism corrected calibration like so $$cal_{corr}(p)=cal_{app}(p)-<Optimism(p)>$$
Important note: The procedure above is inspired by Harrell's work and my discussion with him, but all errors are mine alone.
