When to use smoothing of the predictor in a time dependent incident dynamic ROC analysis, and is there any arguments not to use smoothing?

Question

In the article “Estimation of incident dynamic AUC in practice”, Geloven et al. 2020 (https://www.sciencedirect.com/science/article/pii/S0167947320301869#b10) they describe:
“In recent years, several methods have been proposed to estimate I/D AUC. Each method makes a different choice for how and when it applies the necessary smoothing. Heagerty and Zheng impose the smoothing by assuming a Cox model for the relation between the predictor under evaluation and the event (Heagerty and Zheng, 2005). For each event time point they then estimate the true positive component of the area under the ROC curve based on this Cox model. The smoothing is thus introduced before the actual AUCs are estimated.” From this article I get the impression that smoothing of the predictor is a necessary measure with the aim to dampen the potential fluctuation related to the estimation of time dependent sensitivity: “one of the components of the I/D AUC, for a small event number will in general lead to unstable results. In the most extreme situation, the AUC could jump from 1 to 0 and back between adjacent time points. Therefore, estimation of the I/D AUC requires some degree of smoothing (Saha-Chaudhuri and Heagerty, 2013)” (Also from Geloven et al. )

However, in the example in the “risksetROC-package”, there does not seem to be applied any smoothing, (see image from “rdrr” with script and a plot with abrupt fluctuations in AUC at different time points.):

I have seen at least one article (“Radiomic MRI Phenotyping of Glioblastoma: Improving Survival Prediction” by Bae et al. 2018, https://pubs.rsna.org/doi/10.1148/radiol.2018180200) using the incident dynamic time-dependent ROC analysis, apparently with no smoothing based on the pronounced abrupt fluctuations in the plots (It is not described in further details in methods):

I will also provide an example from some of my own data: Where I first present an example where I have applied the pspline() function on the predictor in the risksetAUC-analysis from the risksetROC analysis, yielding an iAUC of 0.82 for the defined time-period, we can see that there is a high discriminatory performance for the whole time period.:

When we apply the same predictor for the same time period, but without the pspline(predictor)-funtion applied we see in this dataset, that there is not a lot of fluctuation, but there is a more pronounced fall in discriminatory performance in the beginning of the period, which leads to an overall lower iAUC at 0.78 :

The questions are: Is it obligatory to use some kind of smoothing on the predictor in this kind of analysis? If so, are the methods described in the mentioned article (“Radiomic MRI Phenotyping of Glioblastoma: Improving Survival Prediction” by Bae et al. 2018) and in the package description of RisksetROC flawed? Could smoothing lead to an falsely inflated iAUC? (In a situation where there is a real decline in the discriminatory performance of the predictor)?

To sum up: When should we use smoothing of the predictor in a time dependent incident dynamic ROC analysis?

EDIT: I find this interesting, but somewhat confusing. It makes a lot of sense that the smoothing is integrated in the risksetROC()-algorithm. However, I have seen several people more statistical gifted people than me using the pspline()-function on contionous variables in the cox-model from which the linear predictor is derived. An example that comes to mind is Kazuki Yoshida (Epidemiologist/Biostatician at Harvard, I believe) in this blog-post at Datascience+ (https://datascienceplus.com/time-dependent-roc-for-survival-prediction-models-in-r/) Where he fits the cox-model by using the penalized spline function on the continuous variable age.

I therefore assumed that this was the smoothing-step required for correct estimates. I have used this method for several data-set, and the using the pspline(continuous_predictor) in the cox-model yields higher iAUC-values than without. I have also noticed that the concordance index is the unaffected by pspline(), but the corresponding iAUC is lower when not using the pspline():
model<-coxph(so~pspline(continuous_predictor)) Concordance= 0.825 (se = 0.03 ) iAUC: 0.817943

model<-coxph(so~ (continuous_predictor) Concordance= 0.825 (se = 0.03 ) iAUC: 0.7931882 (These are results from real data) I know that there are several ways to calculate the concordance index (Harrels, Uno etc..), And Heagerty describes the iAUC as following: “We show that a global concordance measure is the integral, or weighted average, of time-specific AUC measures.” In the paper “ “Survival Model Predictive Accuracy and ROC Curves” from 2005. (https://onlinelibrary-wiley-com.pva.uib.no/doi/10.1111/j.0006-341X.2005.030814.x). And I have observed in several data-sets that the iAUC is more similar to Harrel`s- c-index when using the pspline() function. And I have interpreted this as the effect of “correctly” applying the pspline to obtain the necessary smoothing for the analysis. But if the predictors are smoothed in the RisksetROC()-algorithm, I am left somewhat confused about what I achieve by using the pspline(), besides apparently improving the discriminatory performance of the model.

Is it the case that the smoothing applies only for the risksetROC, but not for risksetAUC? And smoothing can be applied for risksetAUC by using pspline?

What does the pspline add to the model performance, that is not provided by the RisksetROC-smoothing? The concordance index is according to my understanding measuring the ratio of concordant and disconcordant pairs according to the ranking og event times. This means that if there are high discriminatory performance at earlier times, then the C-index might still be high even if there are lower discriminatory performance at later times (Especially since there often are more events at earlier times than later times in surivival data). Therefore a fall in discriminatory performance, and manifest low discriminatory performance might be masked by a high c-index due to a high overshadowing early discriminatory performance. One of the interesting concepts of I/D-ROC analysis is that it evaluates the predictive performance at each time point of events, and therefore can evaluate better how the discriminatory performance evolves over time, rather than only providing a cumulative summary at each time point like the cumulative/dynamic approach. One of my worries of using a penalized spline in fitting the cox-model is that it will potentially "dampen" a real fall in discriminatory performance. This might be of a concern if a clinician uses a predictive model to evaluate risk of event for a given patient who has survived to timepoint a, defined as in the "riskset", and the model is inaccurate for later times of follow-up, and provides unceratin estimates.

When should the pspline()-function be used, and could it potentially reduce the analysis sensitivity to detect real effect in falling discriminatory performance of the model?

I hope the reason of my confusion makes sense. In the mean time; I will try to read up on smoothing-methods, since I am obviously missing something here. Thanks.

Answer 1

The main need for smoothing is for ROC curves themselves, not in plots of ROC-AUC against time. The several types of smoothing at play in this question might be leading to some confusion.

Without some type of smoothing on censored data you can get very weirdly shaped ROC curves that make AUC calculations meaningless. That's put clearly in the Heagerty et al. paper on what was later called cumulative/dynamic ROC curves based on Kaplan-Meier estimation. With X as the covariate and its cutoff value c varied to construct the ROC curve:

as the conditioning set X > c changes, the redistribution to the right ... of the probability mass associated with censored observations also changes...The changing redistribution can lead to inconsistencies that produce negative probability mass and therefore produce ROC curves that are not monotone.

To overcome that problem, Heagerty et al. used a locally smoothed estimate of the bivariate distribution of covariate and survival. That's one type of smoothing.

Incident/dynamic ROC curves can be derived directly from a Cox model and the data if proportional hazards (PH) hold, as Heagerty and Zheng described (drawing on prior work by Xu and O'Quigley). In this context, I think of a Cox model as directly providing a smoothed estimate of the bivariate distribution of covariate and survival. That's the "Cox" smoothing noted in one of the quotes in the question, implemented as the default method in the risksetAUC() and the underlying risksetROC() functions. Thus smoothing is built into the algorithm by default.

If PH doesn't hold, a time-dependent survival model must be estimated to get incident/dynamic ROC curves with this approach. That's provided by the alternate choices for the method setting in the function, "LocalCox" and "Schoenfeld." As those estimates are done locally in time one needs to specify the effective time span, another type of smoothing.

Then there's the issue of the type of smoothing used to get the linear predictor whose values are varied along the ROC curve. That's yet a further type of smoothing, which seems to be at issue in most of the question: non-linear smooth modeling of a continuous covariate in the underlying Cox model (that in turn provides the smoothing needed for the ROC-curve construction). Insofar as that nonlinear covariate modeling, e.g. with pspline(), improves the Cox model over a linear treatment of the covariate, one might expect better overall AUC.

Even with that smoothing to get reasonable ROC curves, however, you can still get abrupt changes in the incident/dynamic AUC as the times for ROC evaluations are varied--as your plots show. The "incident/dynamic" ROC curves are calculated at event times (that's the "incident" part), so some fluctuations in ROC-AUC over time, depending on the cases experiencing the event at different times, are expected.