In the article “Estimation of incident dynamic AUC in practice”, Geloven et al. 2020 (https://www.sciencedirect.com/science/article/pii/S0167947320301869#b10) they describe:
“In recent years, several methods have been proposed to estimate I/D AUC. Each method makes a different choice for how and when it applies the necessary smoothing. Heagerty and Zheng impose the smoothing by assuming a Cox model for the relation between the predictor under evaluation and the event (Heagerty and Zheng, 2005). For each event time point they then estimate the true positive component of the area under the ROC curve based on this Cox model. The smoothing is thus introduced before the actual AUCs are estimated.” From this article I get the impression that smoothing of the predictor is a necessary measure with the aim to dampen the potential fluctuation related to the estimation of time dependent sensitivity: “one of the components of the I/D AUC, for a small event number will in general lead to unstable results. In the most extreme situation, the AUC could jump from 1 to 0 and back between adjacent time points. Therefore, estimation of the I/D AUC requires some degree of smoothing (Saha-Chaudhuri and Heagerty, 2013)” (Also from Geloven et al. )
However, in the example in the “risksetROC-package”, there does not seem to be applied any smoothing, (see image from “rdrr” with script and a plot with abrupt fluctuations in AUC at different time points.):
I have seen at least one article (“Radiomic MRI Phenotyping of Glioblastoma: Improving Survival Prediction” by Bae et al. 2018, https://pubs.rsna.org/doi/10.1148/radiol.2018180200) using the incident dynamic time-dependent ROC analysis, apparently with no smoothing based on the pronounced abrupt fluctuations in the plots (It is not described in further details in methods):
I will also provide an example from some of my own data: Where I first present an example where I have applied the pspline() function on the predictor in the risksetAUC-analysis from the risksetROC analysis, yielding an iAUC of 0.82 for the defined time-period, we can see that there is a high discriminatory performance for the whole time period.:
When we apply the same predictor for the same time period, but without the pspline(predictor)-funtion applied we see in this dataset, that there is not a lot of fluctuation, but there is a more pronounced fall in discriminatory performance in the beginning of the period, which leads to an overall lower iAUC at 0.78 :
The questions are: Is it obligatory to use some kind of smoothing on the predictor in this kind of analysis? If so, are the methods described in the mentioned article (“Radiomic MRI Phenotyping of Glioblastoma: Improving Survival Prediction” by Bae et al. 2018) and in the package description of RisksetROC flawed? Could smoothing lead to an falsely inflated iAUC? (In a situation where there is a real decline in the discriminatory performance of the predictor)?
To sum up: When should we use smoothing of the predictor in a time dependent incident dynamic ROC analysis?
EDIT: I find this interesting, but somewhat confusing. It makes a lot of sense that the smoothing is integrated in the risksetROC()-algorithm. However, I have seen several people more statistical gifted people than me using the pspline()-function on contionous variables in the cox-model from which the linear predictor is derived. An example that comes to mind is Kazuki Yoshida (Epidemiologist/Biostatician at Harvard, I believe) in this blog-post at Datascience+ (https://datascienceplus.com/time-dependent-roc-for-survival-prediction-models-in-r/) Where he fits the cox-model by using the penalized spline function on the continuous variable age.
I therefore assumed that this was the smoothing-step required for correct estimates.
I have used this method for several data-set, and the using the pspline(continuous_predictor) in the cox-model yields higher iAUC-values than without. I have also noticed that the concordance index is the unaffected by pspline(), but the corresponding iAUC is lower when not using the pspline():
model<-coxph(so~pspline(continuous_predictor)) Concordance= 0.825 (se = 0.03 ) iAUC: 0.817943
model<-coxph(so~ (continuous_predictor) Concordance= 0.825 (se = 0.03 ) iAUC: 0.7931882 (These are results from real data) I know that there are several ways to calculate the concordance index (Harrels, Uno etc..), And Heagerty describes the iAUC as following: “We show that a global concordance measure is the integral, or weighted average, of time-specific AUC measures.” In the paper “ “Survival Model Predictive Accuracy and ROC Curves” from 2005. (https://onlinelibrary-wiley-com.pva.uib.no/doi/10.1111/j.0006-341X.2005.030814.x). And I have observed in several data-sets that the iAUC is more similar to Harrel`s- c-index when using the pspline() function. And I have interpreted this as the effect of “correctly” applying the pspline to obtain the necessary smoothing for the analysis. But if the predictors are smoothed in the RisksetROC()-algorithm, I am left somewhat confused about what I achieve by using the pspline(), besides apparently improving the discriminatory performance of the model.
Is it the case that the smoothing applies only for the risksetROC, but not for risksetAUC? And smoothing can be applied for risksetAUC by using pspline?
What does the pspline add to the model performance, that is not provided by the RisksetROC-smoothing? The concordance index is according to my understanding measuring the ratio of concordant and disconcordant pairs according to the ranking og event times. This means that if there are high discriminatory performance at earlier times, then the C-index might still be high even if there are lower discriminatory performance at later times (Especially since there often are more events at earlier times than later times in surivival data). Therefore a fall in discriminatory performance, and manifest low discriminatory performance might be masked by a high c-index due to a high overshadowing early discriminatory performance. One of the interesting concepts of I/D-ROC analysis is that it evaluates the predictive performance at each time point of events, and therefore can evaluate better how the discriminatory performance evolves over time, rather than only providing a cumulative summary at each time point like the cumulative/dynamic approach. One of my worries of using a penalized spline in fitting the cox-model is that it will potentially "dampen" a real fall in discriminatory performance. This might be of a concern if a clinician uses a predictive model to evaluate risk of event for a given patient who has survived to timepoint a, defined as in the "riskset", and the model is inaccurate for later times of follow-up, and provides unceratin estimates.
When should the pspline()-function be used, and could it potentially reduce the analysis sensitivity to detect real effect in falling discriminatory performance of the model?
I hope the reason of my confusion makes sense. In the mean time; I will try to read up on smoothing-methods, since I am obviously missing something here. Thanks.