# Interpreting calibration plots

This is some kind of really specific/multidisciplinar question, so maybe this is not the appropiate place to expose this, as it's quite code-heavy. If that's the case, please inform me!

I have fitted a survival model. Explicitly a Cox model. I am using Harrell's rms package (in R).

So far I've done the following. I did a multiple imputation. After that using the fit.mult.impute function I fitted the model. After that I went to an internal bootstrap validation. After reading this post answer by Dr. Harrell I single imputed my data (using mice because I'm more familiar with that) and obtained the following:

As you can see I'm not specifying a timepoint (u=X), but if I do, I get quite the same results.

Now I wanted to do a calibration plot using calibrate from rms itself. I did so but the graphs I got confuse me. First of all I have to set a timepoint. In which I specify 3 months (so u=3)

calis <- calibrate(cph.VIO, method="boot", dxy=T, B=1000, u=3) plot(calis, las=1)

As you can see, the plot if way off. And the two lines (observed + corrected) are basically the same, I am not seeing any correction, even if the validation output shows some optimism-correction.

Things get worse as I try different timepoints. If I was to intreptret this graph as it is right now, I would say model is underestimating probabilities (0.4 predicted is a 0.95 observed), and I would think is a matter of the time point. Because my model ranges from 0 to up to 30 months, I would expect to be more "accurate" on 15-20 months survival probabilities. But the graph is very similar to this one:

Yeah, it is "much better" but still corrected and observed lines overlap and I don't think this is an acceptable result.

So, what do you think is the problem here?

I don't know if I have some code issues or is a more deep problem with all of this. If I was to judge only for the validation output I would have said I had a pretty decent model with a c-stat over .7 index corrected. But looking at the calibration plot makes me think something is not quite right with this. There is really a difference from the plot compared to my output? Do you spot any problems aside of the calibration? What do you think I can try?

As an extra:

If I'm not mistaken, bootstrap is supposed to correct for overfitting. As it seems, my bootstrap is not correcting anything. Does that mean that I have to throw everything? Is my model so badly overfitted that no correction can be done?

Thank you very much!

EDIT: As stated by prof.Harrell, it seems like cph must also include the time point. The model which gave the above output was this

cph.VIO<-cph(Surv(dat$os_time,dat$dead_str)~num+rati+baseline+LDH+he+wellf , data=dat,x=T,y=T,surv=TRUE);cph.VIO

I fitted a new model with:

time.inc=3 and repeteated validation and calibration with u=3, this is the output graph:

Even if far from perfect, I'm now, not overly concerned about overestimating low probabilites on low time-points (in survival I think this would be "acceptable" and, at 7 months, the two lines follow near perfectly the ideal one). Still I'm a little bit concerned on the overlapping (even if the boostrat-correction was little) I would expect more of a separation between observed and optimism-corrected lines.

Just for completion purposes I attach the histogram prof.Harrell, asked for.

• Well stated question. It's acting as if you have a mismatch between u= and the units of measurement in the Surv object. Add to the code the actual call to cph and add a histogram of the failure/censoring times. – Frank Harrell Aug 12 at 11:08
• Thanks for your answer prof.Harrell!! Your comment gave me the answer. I was not using the time.inc on the cph model. Hence the problem. I will update the post with my new calibration just in case. – N00b Aug 12 at 11:50