This is some kind of really specific/multidisciplinar question, so maybe this is not the appropiate place to expose this, as it's quite code-heavy. If that's the case, please inform me!
I have fitted a survival model. Explicitly a Cox model.
I am using Harrell's
rms package (in R).
So far I've done the following.
I did a multiple imputation. After that using the
fit.mult.impute function I fitted the model.
After that I went to an internal bootstrap validation. After reading this post answer by Dr. Harrell I single imputed my data (using
mice because I'm more familiar with that) and obtained the following:
As you can see I'm not specifying a timepoint (
u=X), but if I do, I get quite the same results.
Now I wanted to do a calibration plot using
rms itself. I did so but the graphs I got confuse me. First of all I have to set a timepoint. In which I specify 3 months (so
calis <- calibrate(cph.VIO, method="boot", dxy=T, B=1000, u=3)
As you can see, the plot if way off. And the two lines (observed + corrected) are basically the same, I am not seeing any correction, even if the validation output shows some optimism-correction.
Things get worse as I try different timepoints. If I was to intreptret this graph as it is right now, I would say model is underestimating probabilities (0.4 predicted is a 0.95 observed), and I would think is a matter of the time point. Because my model ranges from 0 to up to 30 months, I would expect to be more "accurate" on 15-20 months survival probabilities. But the graph is very similar to this one:
So, what do you think is the problem here?
I don't know if I have some code issues or is a more deep problem with all of this. If I was to judge only for the validation output I would have said I had a pretty decent model with a c-stat over .7 index corrected. But looking at the calibration plot makes me think something is not quite right with this. There is really a difference from the plot compared to my output? Do you spot any problems aside of the calibration? What do you think I can try?
As an extra:
If I'm not mistaken, bootstrap is supposed to correct for overfitting. As it seems, my bootstrap is not correcting anything. Does that mean that I have to throw everything? Is my model so badly overfitted that no correction can be done?
Thank you very much!
EDIT: As stated by prof.Harrell, it seems like
cph must also include the time point. The model which gave the above output was this
cph.VIO<-cph(Surv(dat$os_time,dat$dead_str)~num+rati+baseline+LDH+he+wellf , data=dat,x=T,y=T,surv=TRUE);cph.VIO
I fitted a new model with:
time.inc=3 and repeteated validation and calibration with
u=3, this is the output graph:
Even if far from perfect, I'm now, not overly concerned about overestimating low probabilites on low time-points (in survival I think this would be "acceptable" and, at 7 months, the two lines follow near perfectly the ideal one). Still I'm a little bit concerned on the overlapping (even if the boostrat-correction was little) I would expect more of a separation between observed and optimism-corrected lines.
Just for completion purposes I attach the histogram prof.Harrell, asked for.