Funky calibration plot from cph Cox regression - do I have to worry?

Question

I have fitted a Cox regression with {rms}, N=3340, events = 2617, set time.inc at 19, the maximal time in my dataset. I used B=400 validation, followed by a B=400 calibration at u=19. My variables are log transformed, where needed, otherwise continuous and factors mixed. I have looked for interactions, none helped with the fit or with concordance (my chosen fit index, as the end product is a ranking). Concordance is ~0.750, only nonlinearity in model is a cubic spline on one of the variables. Median survival is 4 days, so not a lot of data in the tail. (I unfortunately cannot tell more about my data.)

Calibrating the model gave me this plot. If I understand correctly, this is much like a QQplot, so actual surviving is higher, than what I predict, depending on the predicted rate.

What is interesting (at least to me, I'm a dum-dum), is that validating and calibrating for a slightly shorter timeframe (e.g. 15 days instead of 19) gives a way, WAY better fit, which is understandable, but it gives me the nerves, as there is a clear turnaround in the plots at t = 17.

Do I need to worry about this plot? My concordance is 0.75, pretty good for me, especially in my domain, but does this hint at missing variables, nonlinear effects or interactions? I have found no examples on how to interpret this other than what I wrote above. Otherwise model is "well behaving", no surreal coefficients, convergence problems or other anomalies - only that PH assumptions are not met, but so far what I've read about this is that I shouldn't make a big deal out of it.

Answer 1

Questions about the R rms package are slightly more appropriate for https://discourse.datamethods.org/t/rms-discussions but here are some answers:

You did not specify the number of observations for which you have follow-up of 19d or better, so I'll assume this is relatively small. In that case, you are doing too much extrapolation. Try using a time horizon for which there are at least 200 observations followed longer than that. You did not state the mode of validation but I assume it is internal validation using the Efron-Gong optimism bootstrap. You have not followed the procedures in RMS that recognize all sources of model uncertainty. With bootstrap or cross-validation using rms::validate and calibrate you either need to fully pre-specify the model, or to have the bootstrap procedure repeat all potentially model-changing steps afresh for each bootstrap resample. This is only implemented in calibrate and validate for fast backwards stepdown. In other words, bootstrapping can easy account for removal of some apparently unimportant predictors. But it is not programmed to account for playing with the number of knots, etc. The only easy way out is to validate the fullest model that you entertain, since that will properly penalize you for the larger analysis degrees of freedom.

When you ran calibration assessments for shorter time horizons you still have enough miscalibration to worry about. Use smoothed scaled Schoenfeld residual plots to check the proportional hazards assumption for predictors, and also check assumptions of an accelerated failure time model (see the case study on that in RMS) to see if that model fits better than the Cox model.